Benchmarks & Evals

Running niki eval against seeded defect datasets and regression suites.

Benchmarks & Evaluations

Test NIKI against seeded defect datasets using niki eval.

Dataset Format (evals/dataset.toml)

Evaluations use dataset files with [[cases]] entries and a nested seeded-defect table:

[[cases]]
id           = "defect-001"
description  = "Add pagination to GET /items with limit and offset."
replay_dir   = "fixtures/defect-001"
difficulty   = "medium"

[cases.seeded_defect]
label           = "Missing pagination bounds"
category        = "boundary"
keyword         = "pagination"
expected_caught = true

Running Evaluations

# Replay 5 recorded cases (default; no API key required)
niki eval --dataset evals/dataset.toml --limit 5

# Run live models and write reports under an output directory
niki eval --dataset evals/dataset.toml --live --out .niki/eval

--live mutates the target working tree without creating a branch or commit, and refuses a dirty tracked tree unless --allow-dirty is also passed.