Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Test values against a model

This guide checks single values against a model, generates boundary and invalid values for tests, and checks a log of recorded values in one run. It uses the thermostat.sdf.json from Install and try a model, whose targetTemperature accepts 5 to 30 °C in half-degree steps.

Check one sample

A sample is one JSON value for one definition. Save 21.5 in comfortable.json and check it against the target temperature. --kind property and --pointer select the definition, and the trailing argument supplies the model file:

$ esdieff sample --document thermostat.sdf.json --kind property --pointer /sdfObject/Thermostat/sdfProperty/targetTemperature --sample comfortable.json thermostat.sdf.json
Esdieff: passed
Complete: true
Errors: 0; warnings: 0; notices: 0; incomplete checks: 0
Scope: selected
...
Selected property in model 1 at "/sdfObject/Thermostat/sdfProperty/targetTemperature"
Sample 2: admitted; context available
...

Sample 2: admitted is the verdict on the value. Now save 35 as too-hot.json:

$ esdieff sample --document thermostat.sdf.json --kind property --pointer /sdfObject/Thermostat/sdfProperty/targetTemperature --sample too-hot.json thermostat.sdf.json
Esdieff: failed
Complete: true
Errors: 1; warnings: 0; notices: 0; incomplete checks: 0
...
Sample 2: rejected; context available
...
sample.bound: Sample violates an exact numerical bound
source 2 ""
  related effective 1 "/sdfObject/Thermostat/sdfProperty/targetTemperature/maximum"
  related source 1 "/sdfObject/Thermostat/sdfProperty/targetTemperature/maximum"
...

The reports are trimmed to the lines about the sample. A rejected sample is a failed check with exit status 1, and the finding names the rule it broke and where that rule is defined. The other kinds are data for an sdfData definition, action_input, action_output and event_output, which select the data an action or event carries. Numbers compare exactly, so 30.0 and 30 are the same value and 0.1 is never rounded.

Generate values for every rule

esdieff generate builds values from the model’s rules, so your tests get expectations without hand-written payloads. Select one definition with --kind and --pointer, or leave them out to cover every property, action and event in the document:

$ esdieff generate --document thermostat.sdf.json --kind property --pointer /sdfObject/Thermostat/sdfProperty/targetTemperature thermostat.sdf.json
Generated values esdieff.generation/1: passed
Values: 2 admitted, 8 rejected, 0 undetermined
Descriptions: 1

property /sdfObject/Thermostat/sdfProperty/targetTemperature: complete
  rejected     null          null (sample.null)
  admitted     minimum       5
  admitted     maximum       30
  rejected     below_minimum 4 (sample.bound)
  rejected     above_maximum 31 (sample.bound)
  rejected     not_multiple  5.25 (sample.multiple)
  rejected     wrong_type    "" (sample.type)
  rejected     wrong_type    true (sample.type)
  rejected     wrong_type    {} (sample.type)
  rejected     wrong_type    [] (sample.type)

Each value aims at one target, named in the middle column: the two bounds, a value just outside each, a value off the step, and one wrong type of each kind. Esdieff then checks every value against the model, and the first column is the result of that check, not the intention. The two agree here, but a model that allows null would list the null target as admitted. For a map, generation also builds a value missing each required entry, and a wrong value for each entry, as the full thermostat output in the getting-started guide shows for the configure action.

Generation is deterministic. The same model and selection always give the same values in the same order, so the output can live under version control. Save the values as test vectors with --format jsonl, which writes one JSON object per line:

$ esdieff generate --document thermostat.sdf.json --kind property --pointer /sdfObject/Thermostat/sdfProperty/targetTemperature --format jsonl thermostat.sdf.json > vectors.jsonl
$ head -3 vectors.jsonl
{"kind":"property","pointer":"/sdfObject/Thermostat/sdfProperty/targetTemperature","value":null,"expected":"rejected","label":"null"}
{"kind":"property","pointer":"/sdfObject/Thermostat/sdfProperty/targetTemperature","value":5,"expected":"admitted","label":"minimum"}
{"kind":"property","pointer":"/sdfObject/Thermostat/sdfProperty/targetTemperature","value":30,"expected":"admitted","label":"maximum"}

Each line carries the selector, the value and its settled result as expected. Feed the values to the device or app under test and assert that it accepts the admitted ones and refuses the rejected ones. --format json gives the same values inside the full report. The generation contract lists every target.

Check a log of recorded values

esdieff trace checks a recorded trace, a JSON Lines file of values your device or app actually produced, in one run. Each line names a definition the way the generated vectors do, carries the value and may say what you expected. Save this as readings.jsonl:

{"kind":"property","pointer":"/sdfObject/Thermostat/sdfProperty/targetTemperature","value":21.5,"label":"evening"}
{"kind":"property","pointer":"/sdfObject/Thermostat/sdfProperty/targetTemperature","value":21.3,"label":"app bug"}
{"kind":"property","pointer":"/sdfObject/Thermostat/sdfProperty/currentTemperature","value":null,"label":"sensor offline"}
{"kind":"action_input","pointer":"/sdfObject/Thermostat/sdfAction/configure","value":{"targetTemperature":31,"mode":"heat"}}
{"kind":"event_output","pointer":"/sdfObject/Thermostat/sdfEvent/alarm","value":{"temperature":85,"reason":"overheat"},"expected":"rejected"}

$ esdieff trace --document thermostat.sdf.json --trace readings.jsonl thermostat.sdf.json
Recorded trace esdieff.trace/1: failed
Entries: 3 passed, 2 failed, 0 undetermined
Admissions: 2 admitted, 3 rejected, 0 undetermined

line 2 property "/sdfObject/Thermostat/sdfProperty/targetTemperature": failed
  label "app bug"
  admission rejected
  error sample.multiple at "/value" bytes 206..210: Sample is not an exact multiple of the positive step

line 4 action_input "/sdfObject/Thermostat/sdfAction/configure": failed
  admission rejected
  error sample.bound at "/value/targetTemperature" bytes 459..461: Sample violates an exact numerical bound

Only the failed lines are listed, each with its line number, the byte range of the offending part of the line and the rule it broke. Line 1 passed because 21.5 is admitted. Line 3 passed because the current temperature is nullable. Line 5 passed too: the alarm’s temperature of 85 is above its maximum of 80, and the line said that rejection was expected, so an expected rejection counts as a pass. Lines 2 and 4 failed because nothing was expected and the values were rejected.

Each line is checked on its own, so one malformed line fails only itself and the rest of the file still runs. The model is composed once, which keeps a trace of a hundred thousand values fast. The trace contract describes the line format and the limits.

Replay generated vectors against a new revision

Generated vectors are a recorded trace, so esdieff trace replays them. Against the model they were generated from, every line passes:

$ esdieff trace --document thermostat.sdf.json --trace vectors.jsonl thermostat.sdf.json
Recorded trace esdieff.trace/1: passed
Entries: 10 passed, 0 failed, 0 undetermined
Admissions: 2 admitted, 8 rejected, 0 undetermined

Against a revision that raised the maximum to 35 and dropped the half-degree step, exactly the vectors whose result changed fail:

$ esdieff trace --document revised.sdf.json --trace vectors.jsonl revised.sdf.json
Recorded trace esdieff.trace/1: failed
Entries: 8 passed, 2 failed, 0 undetermined
Admissions: 4 admitted, 6 rejected, 0 undetermined

line 5 property "/sdfObject/Thermostat/sdfProperty/targetTemperature": failed
  label "above_maximum"
  admission admitted, expected rejected
  error trace.expected at "/expected" bytes 648..658: The definite admission differs from the Expected admission

line 6 property "/sdfObject/Thermostat/sdfProperty/targetTemperature": failed
  label "not_multiple"
  admission admitted, expected rejected
  error trace.expected at "/expected" bytes 791..801: The definite admission differs from the Expected admission

This is a quick way to see what a model change does to values you already rely on. Compare two revisions explains the same change from the app’s point of view, and Repeat checks and share reports saves sample checks in a plan to run again.