Reports and formats

Every command that evaluates something produces a versioned JSON document. The terminal output is a summary of the same data: --format json prints the full document to stdout, and --output FILE writes it.

Document	Written by	Version field
Report	test, fuzz, test --all, report	report_version: "0.1"
Diff report	diff	diff_version: "0.1"
Replay report	replay --format json	replay_version: "0.1"
Production report	check	check_version: "0.1"

All of them carry decguard_version, created_at, status (pass, warn or fail) and exit_code. Their formats are versioned independently of the package: a change in meaning comes with a version bump and a changelog entry.

Report

The report of decguard test and decguard fuzz.

Field	Content
report_version, decguard_version, created_at	format and tool provenance
mode	test (golden gates), fuzz (properties) or all (both)
status, exit_code	pass / warn / fail, and 0 / 0 / 1
contract	name, type, labels, schema_version, hash (sha256:), path
dataset	path, hash (sha256: of the file bytes), n_cases, n_labeled
backend	name, provider, model, model_version, details (redacted provider details)
health	healthcheck status (ok / unhealthy / unknown) and detail
evaluation, requirements, warnings	the settings and gates used
metrics	counts, classification, calibration (with reliability bins), selective, latency
checks	one entry per gate: gate, level, metric, comparison, threshold, value, status, message
failures	failing examples (errors and wrong answers) with their inputs
results	every case: case_id, input, expected, metadata, and either a result or an error
properties	fuzz/all only: see below

A check entry:

{
  "gate": "min_accuracy",
  "level": "requirement",
  "metric": "accuracy",
  "comparison": ">=",
  "threshold": 0.9,
  "value": 1.0,
  "status": "pass",
  "message": "accuracy 1 >= 0.9"
}

A result entry holds the normalized DecisionResult; an error entry holds its kind and message.

properties

Present in fuzz and all reports:

Field	Content
seed	the transformation seed used
fuzz, config	the fuzz settings and every property's settings
summaries	per property: counts of transformed, skipped, errored and evaluated cases; violations and flips with their rates; max TV, JS and deltas
pairs	every transformed case: id (<property>/<case id>/<sample>), steps, input, presentation (option shown → contract label), result or error, comparison, violations, and a reduced minimal example when minimized
skipped	cases a property could not transform, with the reason

Integrity on load

Loading a report — with report, diff or replay — re-verifies it completely:

• every stored result has probabilities over exactly the contract labels, in canonical order, summing to 1 within the recorded probability_tolerance; selected is the argmax (ties to the first label) and confidence its probability;
• each case has exactly one of result or error;
• transformed cases are checked the same way, and their comparisons and violations must match what their results and the recorded property settings imply;
• dataset counts, aggregate metrics, failure examples, property summaries, checks, status and exit code are recomputed and must agree with the stored values.

A tampered or corrupted report is rejected with exit code 2.

Re-evaluation

Because results holds every normalized decision, decguard report report.json --contract edited.yaml recomputes metrics and re-applies the edited gates offline. Recomputation from the same results is bit-identical. For fuzz and all reports it also re-applies edited property limits and levels to the stored transformed cases; transformations are not regenerated.

Diff report

Written by decguard diff.

Field	Content
contract	the decision compared
baseline, candidate	provenance of each run: path, created_at, mode, contract_hash, dataset, backend
same_dataset	whether both runs used the same dataset file (by hash)
evaluation	settings used to recompute metrics on matched cases
counts	cases per run, matched, only in one run, compared; answer flips, newly wrong/correct; newly errored, recovered, still errored
shift	answer flip rate; mean and max TV distance; mean JS divergence; mean, mean absolute and max absolute confidence delta
metrics	per metric: baseline, candidate, delta
segments	per segment: key, value, size, accuracy before and after, flip rate
checks	regression gates, same shape as report checks
changes	each flipped, newly errored or recovered case with its input, answers and confidences

Replay report

Printed by decguard replay --format json.

Field	Content
source	the report replayed
contract_path, contract_hash, contract_changed	the contract used, and whether it differs from the one recorded
backend	the backend replayed against
outcomes	per replayed failure: id, property, case_id; regenerated (the seed regenerates the stored transformation; null for non-deterministic paraphrase providers); the new original result or original_error; the re-judged examples (transformed and reduced); and reproduced

Production report

Written by decguard check.

Field	Content
contract, evaluation	the decision and settings used
config	the production section: segments and gates
dataset	path, hash, n_records, n_outcomes, n_correctness
baseline	source (dataset or report), path, hash, n_records and metrics, or null
metrics	counts, outcomes, calibration, confidence (with histogram), threshold, routing, latency_ms, cost
drift	confidence_tv_distance, confidence_mean_delta, accuracy_drop, ece_increase (null without a baseline)
segments	per metadata group: key, value, metrics, checks
checks	aggregate and segment checks
records	every normalized record

A production report can be passed back as --baseline to the next decguard check; its records are validated again against the contract when it is loaded.

Input formats

Format	Used by	Reference
Decision Contract (YAML/JSON)	every command	Decision Contract
Golden dataset (JSONL/JSON)	validate, test, fuzz	Golden datasets
Production records (JSONL/JSON)	check	Record schema
Paraphrase file (JSONL)	paraphrase with provider: file	Paraphrase providers
decguard.http/0.1 request/response	http backend	HTTP protocol
System One request/answer	systemone backend	Wire format

Status semantics and exit codes are described in Requirements and warnings and the CLI reference.