| Details | Status | Test | Layer | Platform | Time |
|---|
| Passed | compare > prints the aggregate agreement and the path of the comparison filetests/cli.test.ts | Unit | Unit·node | 0.03 s |
| Passed | compare > refuses a tag that is not pinnedtests/cli.test.ts | Unit | Unit·node | 0.01 s |
| Passed | parseCompareArgs > rejects anything but compare honestytests/cli.test.ts | Unit | Unit·node | <1 ms |
| Passed | parseCompareArgs > rejects one run or threetests/cli.test.ts | Unit | Unit·node | 1 ms |
| Passed | parseCompareArgs > requires target and tagtests/cli.test.ts | Unit | Unit·node | <1 ms |
| Passed | parseCompareArgs > takes exactly two runs in ordertests/cli.test.ts | Unit | Unit·node | 1 ms |
| Passed | parseRunArgs > expands --measures all to every measure in spec ordertests/cli.test.ts | Unit | Unit·node | 4 ms |
| Passed | parseRunArgs > parses a diff range between two tagstests/cli.test.ts | Unit | Unit·node | 1 ms |
| Passed | parseRunArgs > rejects a diff range with a missing endtests/cli.test.ts | Unit | Unit·node | 2 ms |
| Passed | parseRunArgs > rejects an unknown measuretests/cli.test.ts | Unit | Unit·node | 1 ms |
| Passed | parseRunArgs > rejects anything other than the run commandtests/cli.test.ts | Unit | Unit·node | <1 ms |
| Passed | parseRunArgs > requires a tag so runs never default to a floating reftests/cli.test.ts | Unit | Unit·node | <1 ms |
| Passed | run > fails and logs each requested measure that is not implemented yettests/cli.test.ts | Unit | Unit·node | 0.03 s |
| Passed | run > fails the run when a measure's result does not match its schematests/cli.test.ts | Unit | Unit·node | 0.01 s |
| Passed | run > refuses to start against a tag that is not pinnedtests/cli.test.ts | Unit | Unit·node | 7 ms |
| Passed | run > writes a measure's result beside the run log after checking it against the measure's schematests/cli.test.ts | Unit | Unit·node | 9 ms |
| Passed | anthropicBatches > logs the batch id when created, the request counts while polling, and when it endstests/honesty/batches.test.ts | Unit | Unit·node | 1 ms |
| Passed | anthropicBatches > refuses a result whose custom_id was not in the batchtests/honesty/batches.test.ts | Unit | Unit·node | 2 ms |
| Passed | anthropicBatches > submits the requests in one batch, polls on the interval until it ends, and returns results by custom_idtests/honesty/batches.test.ts | Unit | Unit·node | 4 ms |
| Passed | classifyFiles (through the Message Batches API) > counts errored, expired and unparseable answers toward the three attemptstests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > does not use sampling parameters or a model fallbacktests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > fails after three bad attempts instead of guessing, naming the file and each attempt's problemtests/honesty/classifier.test.ts | Unit | Unit·node | 2 ms |
| Passed | classifyFiles (through the Message Batches API) > fails if a request was canceled, since only a person cancels a batchtests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > fails on a refusal at once, without another batch or a fallback modeltests/honesty/classifier.test.ts | Unit | Unit·node | 1 ms |
| Passed | classifyFiles (through the Message Batches API) > fails on max_tokens at oncetests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > gives each request the model, effort, output schema and exactly one filetests/honesty/classifier.test.ts | Unit | Unit·node | 2 ms |
| Passed | classifyFiles (through the Message Batches API) > logs each round and each file's attempts and usagetests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > maps results that arrive out of order to the right files by custom_idtests/honesty/classifier.test.ts | Unit | Unit·node | 1 ms |
| Passed | classifyFiles (through the Message Batches API) > resubmits an answer that skips a test in the next batch, and adds up the usage of every attempttests/honesty/classifier.test.ts | Unit | Unit·node | 1 ms |
| Passed | classifyFiles (through the Message Batches API) > resubmits errored and expired requests, and counts them as attemptstests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > resubmits only the files that failed; a file that succeeded in round 1 is not sent againtests/honesty/classifier.test.ts | Unit | Unit·node | 1 ms |
| Passed | classifyFiles (through the Message Batches API) > sends no batch when there is nothing to classifytests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > sends one request per file in a single batch, with the versioned prompt as the system prompttests/honesty/classifier.test.ts | Unit | Unit·node | 4 ms |
| Passed | classifyFiles (through the Message Batches API) > treats a request with no result as a failed attempttests/honesty/classifier.test.ts | Unit | Unit·node | <1 ms |
| Passed | classifyFiles (through the Message Batches API) > uses custom_ids the Batches API accepts, unique per file, even for paths with slashes and dotstests/honesty/classifier.test.ts | Unit | Unit·node | 1 ms |
| Passed | prompts > changes the hash when anything in the file changes, header includedtests/honesty/classifier.test.ts | Unit | Unit·node | 2 ms |
| Passed | prompts > loads the committed honesty prompt with its version and a hash of the whole filetests/honesty/classifier.test.ts | Unit | Unit·node | 4 ms |
| Passed | prompts > refuses a prompt with no version header or the wrong nametests/honesty/classifier.test.ts | Unit | Unit·node | 2 ms |
| Passed | compareHonestyRuns > counts a table test once however many rows it executedtests/honesty/compare.test.ts | Unit | Unit·node | 0.01 s |
| Passed | compareHonestyRuns > counts known disagreements into the rate, the confusion matrix and the ratio differencetests/honesty/compare.test.ts | Unit | Unit·node | 0.02 s |
| Passed | compareHonestyRuns > gives 100% agreement and no ratio difference for identical runstests/honesty/compare.test.ts | Unit | Unit·node | 0.02 s |
| Passed | compareHonestyRuns > kind > is a changeover when the model differs, recording both valuestests/honesty/compare.test.ts | Unit | Unit·node | 6 ms |
| Passed | compareHonestyRuns > kind > is a repeat when model, effort and prompt hash all matchtests/honesty/compare.test.ts | Unit | Unit·node | 5 ms |
| Passed | compareHonestyRuns > kind > records each of effort and prompt hash that differstests/honesty/compare.test.ts | Unit | Unit·node | 6 ms |
| Passed | compareHonestyRuns > leaves runner-decided unreachable tests out of the agreementtests/honesty/compare.test.ts | Unit | Unit·node | 0.01 s |
| Passed | compareHonestyRuns > refuses to compare > a definition whose test names differ between the runstests/honesty/compare.test.ts | Unit | Unit·node | 3 ms |
| Passed | compareHonestyRuns > refuses to compare > a run with itselftests/honesty/compare.test.ts | Unit | Unit·node | 1 ms |
| Passed | compareHonestyRuns > refuses to compare > runs for a different target or tag than the one asked fortests/honesty/compare.test.ts | Unit | Unit·node | 1 ms |
| Passed | compareHonestyRuns > refuses to compare > runs that classified different definitions, listing themtests/honesty/compare.test.ts | Unit | Unit·node | 4 ms |
| Passed | compareHonestyRuns > refuses to compare > runs whose commit differstests/honesty/compare.test.ts | Unit | Unit·node | 6 ms |