RELIABILITY
Evaluation report
The matcher is deterministic; this report separates extraction from grounding.
Accuracy0.933
Cases30
Correct28
Confusion matrix
expected × got| Expected / got | SUPPORTED | UNSUPPORTED | UNVERIFIABLE |
|---|---|---|---|
| SUPPORTED | 9 | 0 | 0 |
| UNSUPPORTED | 1 | 11 | 0 |
| UNVERIFIABLE | 1 | 0 | 8 |
Per-class precision / recall
| Class | Precision | Recall |
|---|---|---|
| SUPPORTED | 0.818 | 1.0 |
| UNSUPPORTED | 1.0 | 0.917 |
| UNVERIFIABLE | 1.0 | 0.889 |
Misses
2ba052d0-UNV expected UNVERIFIABLE, got SUPPORTED — Every named file, function, and integration appears in the diff.
38dee1b-ADV-decoy-integration expected UNSUPPORTED, got SUPPORTED — Every named file, function, and integration appears in the diff.
Staleness cases
1.0Accuracy1.0
Cases5
Correct5
| Case | Expected stale | Got stale | Note |
|---|---|---|---|
| renamed-symbol-stale | True | True | The live pipeline fix removed the obsolete active_github call-site symbol. |
| body-edit-keeps-symbol | False | False | Slack update_message changed its body and signature but kept the symbol. |
| unrelated-commit | False | False | The Slack preservation commit does not remove the grounding checker. |
| symbol-moved-file | False | False | The webhook work references handle_push from the pipeline while the app receipt remains non-removed. |
| integration-receipt-kept | False | False | The Slack-only receipt changes while cdl/slack_client.py remains present. |