Metamorphic properties

Golden tests check answers to fixed inputs. A metamorphic property checks how the answer behaves when the input changes in a controlled way, so it needs no extra labels and catches failures a golden set cannot see.

Most properties are invariances: listing the options in a different order, formatting the labels differently, adding an irrelevant sentence or extra whitespace must not change the decision. Two are not: inversion expects a yes/no answer to swap, and monotonic expects a score to move in one direction.

Properties run with decguard fuzz (properties only) or decguard test --all (golden gates and properties in one report). How runs are seeded, minimized and replayed is described in Fuzzing and replay.

How a property is checked

1. Each dataset case is decided as is: the original.
2. The property derives one or more transformed cases from it, deterministically from the seed.
3. Each transformed case is sent to the same backend.
4. The two probability distributions are compared, and the comparison is checked against the property's limits.

Everything is recorded in the report: the transformation steps, what was sent, the result (mapped back to contract labels), the comparison and any violation.

Option presentation is separate from the input. option_order and label_format do not edit the input text; they change how the labels are presented to the model. Backends receive the labels in the order and form shown (request.decision.labels in Python, decision.labels over HTTP, the question's criteria for System One) and answer with probabilities keyed by those labels. DecGuard maps them back to the contract labels before comparing.

Configuring properties

fuzz:
  seed: 42

properties:
  option_order:
    samples: 3
    max_tv_distance: 0.03  # per transformed case
    max_flip_rate: 0       # share of transformed cases whose answer changes
  irrelevant_context:
    insertions: 2
    max_tv_distance: 0.05
    level: warning         # requirement (default) fails the run; warning only warns

A property is enabled by being present; enabled: false turns it off while keeping its settings. Every property except monotonic needs at least one limit. Contract validation rejects a property that does not apply to the decision type, for example monotonic on a choice decision.

Settings shared by every property

Key	Default	Meaning
enabled	true	turn the property off without deleting it
level	requirement	requirement fails the run on violation; warning turns PASS into WARN
samples	per property	transformed cases generated per dataset case

Limits

Key	Scope	Meaning
max_tv_distance	per transformed case	total variation distance, ½ Σ |p − q|, 0–1
max_js_divergence	per transformed case	Jensen-Shannon divergence in bits, 0–1 (symmetric, finite)
max_abs_delta	per transformed case	largest change of any single label's probability
max_confidence_delta	per transformed case	change of the selected label's confidence
max_violation_rate	aggregate, default 0	share of transformed cases allowed to break a per-case limit
max_flip_rate	aggregate	share of transformed cases whose selected label may change

Every comparison also records the confidence delta, whether the answer flipped and whether the ranking of the labels changed. A failing check shows how far the distribution moved as well as whether the top answer flipped: a model can keep the same answer and still shift 20% of its probability mass because of the option order.

Checks produced

Each enabled property adds up to three checks to the report:

• <name>.max_violation_rate — when a per-case limit is set;
• <name>.max_flip_rate — when set;
• <name>.max_error_rate — always, as a requirement fixed at 0. If the backend fails on a transformed case, or the original case failed, the property was not demonstrated. If no case could be transformed at all, the check fails too.

Property catalog

Property	Decisions	Transformation	Expected
option_order	choice, noul	present the options in another order	invariant
label_format	all	reformat the labels, or use aliases	invariant
irrelevant_context	all	insert irrelevant sentences	invariant
repeated_context	all	insert one irrelevant sentence several times	invariant
whitespace	all	change whitespace	invariant
paraphrase	all	replace the text with a paraphrase	invariant
inversion	noul	negate the input with a literal rewrite	answer swaps
monotonic	score	increase or decrease a numeric input field	score does not move against the direction

option_order

Presents the options as distinct permutations of the contract order. Catches position bias: models that favour whichever option is listed first.

Key	Default
samples	3 distinct permutations (1–100)

label_format

Changes the surface form of the labels.

Key	Default	Meaning
styles	all seven	any of upper (REFUND), lower, capitalized (Refund), lettered (A) refund), numbered (1. refund), quoted ("refund"), bracketed ([refund])
aliases	{}	alternative wordings per label; variant i uses every label's i-th alias
samples	2	variants per case

For noul decisions this covers affirmative/negative surface variation, such as YES / NO, "yes", or aliases like true / false:

properties:
  label_format:
    styles: [upper, capitalized, quoted]
    aliases: {spam: [junk], ham: [legitimate]}
    max_tv_distance: 0.02

irrelevant_context

Inserts sentences that should not affect the decision at sentence boundaries of the text.

Key	Default	Meaning
fragments	eight neutral English sentences	the sentences to insert
insertions	1 (1–10)	sentences inserted per transformed case
samples	2	transformed cases per case

The default fragments are generic ("The weather was mild that day."). Supply your own that are truly irrelevant for your decision; a sentence about shipping is not irrelevant to a refund decision.

repeated_context

Inserts one fragment several times, to catch models distracted by repetition.

Key	Default
fragments	the same defaults as irrelevant_context
repeats	3 (2–20)
samples	1

whitespace

Applies whitespace edits: doubled spaces, line breaks, tabs, leading and trailing blanks.

Key	Default
edits	3 (1–20) edits per transformed case
samples	2

paraphrase

Replaces the text with paraphrases from a provider.

properties:
  paraphrase:
    source: {provider: identity}   # the default
    max_tv_distance: 0.0

Key	Default
source	{provider: identity}
samples	1 (1–20) paraphrases per case

Providers:

• identity — re-asks the same input. No model needed; it catches backends that answer differently each time (non-determinism).

• file — curated paraphrases, deterministic and reviewable: source: {provider: file, path: paraphrases.jsonl}, relative to the contract, with rows {"id": "<case id>", "paraphrases": ["...", "..."]} or {"input": "<text>", "paraphrases": [...]}.

• openai — any OpenAI-compatible chat-completions API:

  source:
    provider: openai
    model: gpt-4o-mini
    url: https://api.openai.com/v1   # default
    api_key_env: OPENAI_API_KEY      # default; null for local servers without a key
    timeout_s: 60                    # default
    temperature: 0.0                 # default

  The run seed is sent with the request, and the model and system fingerprint are recorded. LLM output is not guaranteed to be reproducible, so replay uses the stored paraphrase text.

• Plugins — a decguard.fuzz.paraphrase.Paraphraser subclass registered under the decguard.paraphrasers entry-point group.

A provider failure is recorded as a paraphrase_error for the case.

inversion (noul)

Applies the first matching literal rewrite (find → replace, first occurrence) that negates the input. The answer must swap: the comparison is made with the labels swapped, so max_tv_distance measures the distance from the perfectly inverted answer.

properties:
  inversion:
    rewrites:
      - {find: " is eligible", replace: " is not eligible"}
    max_tv_distance: 0.1
    max_flip_rate: 0      # the answer must invert

Key	Default	Meaning
rewrites	required, at least one	tried in order
samples	1 (1–20)	applicable rewrites used per case

Only use rewrites whose meaning you know inverts the decision. This is a separate property from the invariances, so a model that ignores negation fails here while it may pass everything else.

monotonic (score)

Adds each of deltas to a numeric field of object inputs. The score must not move against direction.

fuzz:
  text_field: text          # other properties edit input.text

properties:
  monotonic:
    field: affected_users
    direction: increasing   # more affected users must not lower the severity
    deltas: [100, 10000]
    max_level_decrease: 0.05
    max_flip_rate: 0        # the selected level must not move against the direction

Key	Default	Meaning
field	required	top-level numeric key of object inputs
direction	increasing	increasing: a larger value must not lower the score; decreasing: the opposite
deltas	[1.0]	positive amounts added to the field, one transformed case each
max_level_decrease	0.0	allowed move of the expected level against the direction, in scale steps

The expected level is Σ index × probability over the ordered levels, so this compares the whole distribution, not only the top answer.

Inputs a property cannot transform

Text transformations need string inputs, or fuzz.text_field to name the key to edit in object inputs. A case a property cannot transform — no text, no matching rewrite, a missing numeric field — is listed under properties.skipped in the report with the reason. It is never dropped silently.