01 · Creation
Kane CLI Hack Day
BUG QUEST
A DAG lost half its rows and Kane said PASS. Bug Quest writes the precise checks that catch it, fixes the code, and has Kane verify the fix.
- Data pipelines fail silently: a DAG shows Success while rows go missing. A short check like "check the pipeline ran" passes, even with bug detection on, because it only verifies the green status.
/ship/bugquest/media/showcase/p66/shot-gCpTRkYC022EAlYrIRW9BKMhajCqPk.png
01 / 05 · BUG QUEST
02 · See it work
Kane CLI Hack Day
See it work
- User names an agent and types a short instruction
- OpenAI expands it into precise Kane objectives with exact expected values
- Kane runs the original instruction and each objective in Chrome
/ship/bugquest/media/showcase/p66/shot-07HGPbvQR7wWfcxUjbdBwLD1OjPTjj.png
02 / 05 · BUG QUEST
03 · The harness
Kane CLI Hack Day
The intelligence behind shipping
Bug Quest takes a short testing instruction and calls the OpenAI API to expand it into precise Kane objectives with exact expected values (e.g. "verify load rows equal extract rows (1000)"). It runs the original instruction and each objective with kane-cli run --agent against an Airflow-style Pipeline Monitor, parses the run_end result (status, result_code, Test Manager link), streams progress live, and scores silent bugs. In our run, the basic instruction passed (code 100) while the precise check failed with code 740 (confirmed bug: 1,000 extracted vs 500 loaded). After the code fix, the same Kane check was re-run with the bug switch still on and passed.
- 01Load step silently drops half the rows.
- →
- 02Expand instruction into precise row and status checks.
- →
- 03Kane fails the check: code 740, 1,000 vs 500.
- →
- 04Fix code, Kane re-checks; pass ships, fail reverts.
Components: run · sealed_evidence · test_manager
03 / 05 · BUG QUEST
04 · Proof
Kane CLI Hack Day
What Kane actually verified
- ○ Team-reportedKane detected a failure: Open the Pipeline Monitor. Verify load task rows equal extract task rows (1000).run 302 · kane_run:kanehack_cli
- ○ Team-reportedKane detected a failure: Open the Pipeline Monitor. Verify load task rows equal extract task rows (1000).run 301 · kane_run:kanehack_cli
- ○ Team-reportedKane detected a failure: Open the Pipeline Monitor. Verify load task rows equal extract task rows (1000).run 261 · kane_run:kanehack_cli
- ○ Team-reportedEvidence linked by the teamcommit 4e19d25bc2 · https://test-manager.lambdatest.com/projects/01M5P9FBYPFDD0T9S75WWNBR8V/test-cases/01M5QY0HYH23K4TS336V56FDM6/dashboard/share/US_F4D3UDAUNBFF7OXBQZFNEZKO6RR1BTP3XJYGCMSBG9ONERVUTJKSSBNG26K7B5UH?type=summary&agentView=true&fqdn=summary-page
- ○ Team-reportedEvidence linked by the teamcommit 4e19d25bc2 · https://test-manager.lambdatest.com/projects/01M5P9FBYPFDD0T9S75WWNBR8V/test-cases/01M5QZGGR673GHG0HHX95JHZM6/dashboard/share/US_6H6J3I8I5EUE676MW4VL6R7GMLVN2MEWAMII2J8YFNC05V0KT30K7VQV34Q3UAT2?type=summary&agentView=true&fqdn=summary-page
Not yet verified: Pipeline is simulated, no real Airflow or warehouse · Auto-fix is verified on the row-mismatch bug only; other bug types not yet tested · Auto-fix edits lib/pipeline.ts only
04 / 05 · BUG QUEST
05 · Outcome
Kane CLI Hack Day
What improved
A DAG lost half its rows and Kane said PASS. Bug Quest writes the precise checks that catch it, fixes the code, and has Kane verify the fix.
1 bug Silent row-mismatch bug caught (baseline 0; source: KR-261; Test Manager runs 01–03; extract 1,000 vs load 500)
1 attempt Auto-fix attempts needed for a Kane-verified fix (baseline 3; source: auto-fix commit e8560fe; Kane re-ran the same check and passed (code 100); rowMismatchBug stayed true)
False alarms after auto-fix (qualitative)
https://github.com/EstrellaAce29/bug-quest · https://d3nl4wy7jovhq0.cloudfront.net/public/p66/6849dfc359e35065374c6f0dbf23fcbc6dbe3d382721743fd4be95119d896338.mp4 · Prashansa Chaudhary
05 / 05 · BUG QUEST