Repeat checks and share reports

This guide saves checks so they run the same way every time, checks a whole directory of models, and brings the results into CI and code review. It assumes Esdieff is installed.

Save checks as a plan

A check plan is a JSON file that names the model and sample files to supply and the cases to run on them. One command then runs every case and reports each outcome by name. The repository’s thermostat-check example includes this plan.json:

{
  "schema": "esdieff.check-plan",
  "version": "1",
  "sources": [
    { "key": "thermostat", "path": "thermostat.sdf.json", "role": "model" },
    { "key": "comfortable", "path": "comfortable.json", "role": "sample" },
    { "key": "too-hot", "path": "too-hot.json", "role": "sample" }
  ],
  "cases": [
    { "name": "Catalogue", "operation": "check", "scope": { "kind": "catalogue" } },
    {
      "name": "Thermostat model",
      "operation": "check",
      "scope": { "kind": "selected", "roots": ["thermostat"] },
      "assessment": "full-baseline",
      "strict_warnings": true
    },
    {
      "name": "Comfortable target",
      "operation": "sample",
      "document": "thermostat",
      "kind": "property",
      "pointer": "/sdfObject/Thermostat/sdfProperty/targetTemperature",
      "sample": "comfortable"
    },
    {
      "name": "Too hot target",
      "operation": "sample",
      "document": "thermostat",
      "kind": "property",
      "pointer": "/sdfObject/Thermostat/sdfProperty/targetTemperature",
      "sample": "too-hot"
    }
  ]
}

Paths are relative to the plan, so the directory moves as a unit. Each case keeps its own scope, assessment level and warning policy. Copy the example directory and run it. The human report prints each case in full, so this command keeps the summary lines:

$ esdieff check --plan plan.json | grep -E '^(Case|Disposition|Admission|Check case summary)'
Case: "Catalogue"
Disposition: passed
Case: "Thermostat model"
Disposition: passed
Case: "Comfortable target"
Disposition: passed
Admission: admitted
Case: "Too hot target"
Disposition: failed
Admission: rejected
Check case summary: {"admission_undetermined":0,"admitted":1,"complete":true,"complete_cases":4,"failed":1,"incomplete_cases":0,"omitted_reports":0,"passed":3,"rejected":1,"request_errors":0,"total":4,"undetermined":0,"verdict":"failed"}

The run exits 1, because the rejected sample is a failed case. A plan has no expected-result field: a sample you intend to reject still fails, which keeps the verdict honest. Keep such samples in a separate plan, or check them with recorded traces, where an expected member turns an intended rejection into a pass. The catalogue-check example is the smallest model-only plan, and a diff case can compare two revisions beside the checks, as Repeatable catalogue checks shows.

Check a directory of models

--directory selects every regular file ending in .sdf.json in a directory, so a check follows the collection as files are added. Run this on the repository’s directory-check example, which holds a device, a shared type, a nested status model and an unfinished draft:

$ esdieff check --directory examples/directory-check --recursive --exclude draft --format json | jq -c '.sources[] | {id, label}'
{"id":1,"label":"examples/directory-check/device.sdf.json"}
{"id":2,"label":"examples/directory-check/nested/status.sdf.json"}
{"id":3,"label":"examples/directory-check/types.sdf.json"}

--recursive descends into subdirectories and --exclude prunes a literal file or subtree relative to each directory. Sample files, plans and a .bak lookalike are not selected. Without the exclusion, the draft’s broken type fails the run:

$ esdieff check --directory examples/directory-check --recursive
Esdieff: failed
Complete: true
Errors: 1; warnings: 0; notices: 0; incomplete checks: 0
...
data.string: A textual data quality must contain a string
source 2 "/sdfData/Unfinished/type"
Diagnostic details omitted: 0

Repeat --directory for several collections, add explicit files beside them, and name a discovered root by its joined path, such as --root examples/directory-check/device.sdf.json. Check a local model collection states the selection rules and limits.

Use JSON reports in automation

--format json writes the same report as a JSON document with a stable schema, for every command. The outcome member is the summary a script needs:

$ esdieff check --format json thermostat.sdf.json | jq '.outcome'
{
  "complete": true,
  "disposition": "passed",
  "errors": 0,
  "incomplete_checks": 0,
  "notices": 0,
  "strict_warnings": false,
  "warnings": 0
}

diagnostics.items holds each finding with its code, message, locations and the RFC section it rests on, and coverage records what was assessed. A plan run has its own schema, esdieff.check-run, whose summary counts the cases:

$ esdieff check --plan plan.json --format json | jq -c '.summary'
{"admission_undetermined":0,"admitted":1,"complete":true,"complete_cases":4,"failed":1,"incomplete_cases":0,"omitted_reports":0,"passed":3,"rejected":1,"request_errors":0,"total":4,"undetermined":0,"verdict":"failed"}

The report contract describes every member and how reports stay within their size limits. Additive fields can appear in later releases, so read the members you need rather than comparing whole documents.

Show findings in code review with SARIF

esdieff check --format sarif and esdieff diff --format sarif write a SARIF 2.1.0 log, the format code-scanning tools such as GitHub’s read to annotate files in a pull request. Each finding becomes a result with its code as the rule, a level and the file, line and column it is about:

$ esdieff check --directory examples/directory-check --recursive --format sarif | jq '.runs[0].results[] | {ruleId, level, message: .message.text, file: .locations[0].physicalLocation.artifactLocation.uri, line: .locations[0].physicalLocation.region.startLine}'
{
  "ruleId": "data.string",
  "level": "error",
  "message": "A textual data quality must contain a string",
  "file": "examples/directory-check/draft/unfinished.sdf.json",
  "line": 3
}

File paths are relative to the directory the command ran in, so run it from the repository root and upload the log with your code-scanning tool’s SARIF step. Acknowledged revision differences appear as suppressed results, as Compare two revisions shows.

Write change notes with Markdown

esdieff diff --format markdown writes a revision comparison as a changelog grouped by breaking, undetermined and compatible changes, ready to paste into release notes or a pull request description. Compare two revisions shows the output.

Use exit statuses

Every command uses the same exit statuses, so a shell or CI step can act on them without parsing a report:

esdieff generate never fails, so it exits 0 or 3. A status of 3 deserves attention in CI as much as 1 does: the model was not shown to be wrong, but it was not shown to be right either.