BUG QUEST

Print view: one slide per page. Choose Save as PDF in the print dialog.

01 · Creation

Kane CLI Hack Day

BUG QUEST

A DAG lost half its rows and Kane said PASS. Bug Quest writes the precise checks that catch it, fixes the code, and has Kane verify the fix.

Data pipelines fail silently: a DAG shows Success while rows go missing. A short check like "check the pipeline ran" passes, even with bug detection on, because it only verifies the green status.

01 / 05 · BUG QUEST

02 · See it work

Kane CLI Hack Day

See it work

1User names an agent and types a short instruction
2OpenAI expands it into precise Kane objectives with exact expected values
3Kane runs the original instruction and each objective in Chrome

02 / 05 · BUG QUEST

03 · The harness

Kane CLI Hack Day

The intelligence behind shipping

Bug Quest takes a short testing instruction and calls the OpenAI API to expand it into precise Kane objectives with exact expected values (e.g. "verify load rows equal extract rows (1000)"). It runs the original instruction and each objective with kane-cli run --agent against an Airflow-style Pipeline Monitor, parses the run_end result (status, result_code, Test Manager link), streams progress live, and scores silent bugs. In our run, the basic instruction passed (code 100) while the precise check failed with code 740 (confirmed bug: 1,000 extracted vs 500 loaded). After the code fix, the same Kane check was re-run with the bug switch still on and passed.

01Load step silently drops half the rows.
02Expand instruction into precise row and status checks.
03Kane fails the check: code 740, 1,000 vs 500.
04Fix code, Kane re-checks; pass ships, fail reverts.

Components: run · sealed_evidence · test_manager

03 / 05 · BUG QUEST

04 · Proof

Kane CLI Hack Day

What Kane actually verified

Team-reportedKane detected a failure: Open the Pipeline Monitor. Verify load task rows equal extract task rows (1000).run 302 · failed · kane_run:kanehack_cli
Team-reportedKane detected a failure: Open the Pipeline Monitor. Verify load task rows equal extract task rows (1000).run 301 · blocked · kane_run:kanehack_cli
Team-reportedKane detected a failure: Open the Pipeline Monitor. Verify load task rows equal extract task rows (1000).run 261 · failed · kane_run:kanehack_cli
Team-reported

05 · Outcome

Kane CLI Hack Day

What improved

A DAG lost half its rows and Kane said PASS. Bug Quest writes the precise checks that catch it, fixes the code, and has Kane verify the fix.

1 bug Silent row-mismatch bug caught (baseline 0 bug; source: KR-261; Test Manager runs 01–03; extract 1,000 vs load 500)
1 attempt Auto-fix attempts needed for a Kane-verified fix (baseline 3 attempt; source: auto-fix commit e8560fe; Kane re-ran the same check and passed (code 100); rowMismatchBug stayed true)
False alarms after auto-fix (qualitative)

https://github.com/EstrellaAce29/bug-quest · https://d3nl4wy7jovhq0.cloudfront.net/public/p66/6849dfc359e35065374c6f0dbf23fcbc6dbe3d382721743fd4be95119d896338.mp4 · Prashansa Chaudhary

Loading deck…