Receipts: what the agent did, and how often a plan was wrong

No login. Every case is fictional. Plans below run live against Qloo and the model when you click; nothing on this page is replayed. Numbers come from bench/results.json, bench/replan.json and bench/audit.json in the repository (written by the scripts named beside them) and from the stored Qloo call logs of this deployment.

The claim

Rekindle is a behavioral-activation planning agent for clinicians. It turns what a depressed patient used to love into a week of real, open, nearby activities, verified with Qloo, approved by the clinician, and re-planned from the patient's own pleasure ratings.

The 30-second path

1. Open the live app and click Plan Maya's week, live. The agent trace lists each Qloo call as it happens; the week arrives in about 20 s.
2. Click In recovery. Alcohol-serving venues are struck in place with their rule (R-ALC …) and those slots are refilled live.
3. Come back here for the six edge states (four with a live start button; refused and empty show on the weeks those start) and the bench, and open /integrations/qloo for every Qloo request behind the latest plans (the key is never shown).

The full click path (swap, approve, rate, re-plan week 2, patient sheet) is in DEMO.md, about 4 minutes.

Receipts in one line

534 Vitest tests in 34 files, no keys needed (npm test) · 60,000 generated verifier cases (3 properties × 20,000, fixed seeds): in none did a PASS outing break a hard constraint · bench run finished 2026-10-07 13:11 UTC · 822.8 s wall clock · 40 Rekindle plans + 20 arm-A model plans · 386 Qloo calls on the plan path · 267 grading calls · 0 errors (bench/results.json).

Reproduce (the real path)

# the product, live: plans call Qloo and the routing model; nothing on this path is replayed
git clone https://github.com/edycutjong/rekindle && cd rekindle && npm ci && npx playwright install chromium
python3 -m venv .venv && . .venv/bin/activate && pip install playwright && playwright install chromium
# ui_smoke spends 2 public plan runs, the E2E 1; past your address's hourly limit or the daily cap, ui_smoke prints "CAPPED at <step>" and exits 3: that is the limit, not a failure (REKINDLE_JUDGE_TOKEN=<token from the judge link> runs it on the judge quota)
python3 scripts/ui_smoke.py https://rekindle.edycu.dev /tmp/shots live
E2E_BASE_URL=https://rekindle.edycu.dev E2E_LIVE=1 npx playwright test e2e/live-plan.spec.ts --project=chromium
# the bench, against a live deployment (needs BENCH_TOKEN and model keys): DEMO.md "Bench (reproduce)"

CI / deterministic, not the product: npm run check and npx playwright test run with no keys. In that mode the app refuses to plan (“No week is invented”) and the E2E suite asserts exactly that.

Honest limitations

1. Not a clinical tool. Clinician support only: no diagnosis, no efficacy claims, no patient logins.
2. A plan is slow: p50 about 17 s from click in the bench, inside our 30 s gate but over the 15 s target.
3. The ember thread is Qloo's attribution, not causation. It is drawn when a week-2 card's top rated input outweighs the case's best original love by at least 0.02; on live re-plans most leads are still ties (see re-plan responsiveness below).
4. The alcohol screen is a deny list. It is measured by the ablation below, but an alcohol-serving venue with none of the tags or keywords would get through.

All seven are in the README (Limitations).

Links

• GitHub repository
• Live app
• DEMO.md (click path + bench)
• JUDGE.md (this page as a file)
• ARCHITECTURE.md (generated from the code)
• Qloo DX report

Six states to check

1. Loading. Pencilled lines fill each day while the agent works; cards are pencilled in as each tool returns.Plan Maya's week, live
2. Refused. Candidates the verifier refused stay in their slot, struck through, with the rule beneath. Toggle in recovery on Maya's sheet to strike bars in place.
3. REVIEW. A candidate the screen cannot decide (e.g. bar seating alone on an in-recovery case) goes to the clinician in a dashed box, never onto the sheet unasked. This runs Maya with in recovery already on, so the screen checks every outing candidate Qloo returns; how many land in REVIEW depends on what Qloo returns that day.Plan Maya in recovery, live Plan Theo (REVIEW in the bench run)
4. Empty. A slot with no verified candidate is left empty with the reason. Also: the clinician's remove.
5. Coverage refusal. When outings within the radius cannot be verified, the sheet says how many and why, and offers a wider radius only with clinician consent.Plan Cal (sparse coverage)
6. Qloo unavailable. Labelled demo: runs Maya with an unreachable Qloo host on purpose. Every slot stays empty with the reason; the model fills nothing on its own. It sends no request and is left out of the receipts.Run the outage demo

Capacity reached (not a state to click): public plan runs share a daily cap and a per-address hourly limit, and the judge link in the Devpost testing instructions has its own daily quota (lib/ratelimit.ts; each plan response carries the cap in x-ratelimit-daily-limit). Past a limit the app says “Capacity reached”, invents no week, and links the latest live runs at Qloo integration. Two stored weeks stay readable without a run: Maya's week 1 and her week 2, re-planned from her ratings.

Bench: named places, checked

scripts/bench.py · 20 seeded profiles (10 in recovery) · finished 2026-10-07T13:11:44+00:00 · against https://rekindle-one.vercel.app

Definition. of the outing venues in each arm's plans across the seeded profiles, the share that are unverifiable, closed at the slot, or serve alcohol to a patient in recovery. Unverifiable: A / B-sym: the stated address does not geocode (Nominatim) or no Qloo /search place within 1 mile matches the name within 300 m. B: the card's entity id is not returned by Qloo /entities. Unknown hours is its own column: Rekindle refuses unknown hours by construction.

Bias. Qloo is the oracle for both arms (the bias the manual audit checks); B-sym runs arm A's address pipeline on Rekindle's own cards. So Rekindle's Qloo-graded rate is low by construction; the manual audit checks the venues against public listings instead. Audit: each sampled venue looked up in public web sources (venue sites, city/tourism listings, review and delivery listings; links in bench/manual_audit.csv) for existence at the stated address, hours covering the slot, and alcohol service on in-recovery cases. Stale (pre-2024) or missing evidence = inconclusive, never guessed. Google Maps was not used. Auditor: the build agent (Claude), not blind: it ran the bench, so arm labels could be inferred from address style; no external human auditor yet. In the same random sample, 4 of the 10 LLM venues the automated checker could not match were real and open (it matches by street address, which misses large venues such as parks), so the one-by-one audit is the headline.

Ablation: is the alcohol screen load-bearing?

B / B- / B-4: distinct places Qloo returned to outing queries that carry an R-ALC deny tag or keyword, counted before R-KIND or any other rule. The last column is what would reach the sheet.

Latency

B plan p50 / p95 (client)

17.43 / 20.61 s · 9.4 Qloo calls avg

B- plan p50 / p95 (client)

18.76 / 20.81 s · 10.1 Qloo calls avg

B-4 plan p50 / p95 (client)

18.01 / 20.86 s · 9.6 Qloo calls avg

arm A model call p50 / p95

7.2 / 8.3 s

Re-plan responsiveness

16/16

N = arm-B week-1 plans with >= 2 verified outings and >= 1 at-home card. PASS = week 2 has >= 1 new entity whose top-weighted explainability input is a >= 7-rated entity, and 0 entities with the <= 3-rated ID. margin = top rekindled weight - best original-love weight on a new card (Qloo's attribution, not a causal claim about the patient). PASS at the thread margin = PASS with at least one such card whose margin is at least 0.02 (thread_margin), the lead the product draws an ember thread for. Excluded: Ruth (week 1 has 1 verified outing(s); the rating rule needs >= 2); Omar (week 1 has 0 verified outing(s); the rating rule needs >= 2); Hana (week 1 has 1 verified outing(s); the rating rule needs >= 2); Cal (week 1 has 0 verified outing(s); the rating rule needs >= 2). Low-rated entities back in week 2: 0. New week-2 cards whose top input is a rekindled rating: 67 of 84 (at the ember thread's own margin, a rating ahead of the best original love by >= 0.02: 50 of those 67, and 16/16 profiles still pass). Median margin over the best original love: 0.0555. Re-plan p50 20.71 s.

Ember threads on this deployment (bench/ember-live.json, scripts/ember-margins.ts, 2026-10-08 08:33 UTC): 7 live re-plans with 3 distinct rating sets (Qloo returns the same week for the same ratings). Of 16 distinct new week-2 cards that name a rating, a rating outweighed the case's best original love on 11. The product draws an ember thread and says “because of” when that lead is at least 0.02 (the bench's margin): 2 cleared it; the other 9 are shown as ties. The previous rule, a lead over the card's next input of any kind, threaded 2 of the same cards. On the bench's 84 new cards (bench/replan.json, same margin), 50 clear it. Qloo splits explainability nearly evenly across signals, so most live re-plan picks are not attributable to one rating. The live sample is small. Qloo's attribution, not causation.

Latest live runs on this deployment

plans

5

Qloo requests

57 (0 non-200)

sent to clinician review

4

agency: cards a model routed (its tool call chose where Qloo looked) · cards with model coaching

29 of 34 · 32 of 34 (the rest: the deterministic route, a kept or refilled card, or a template line; the agent trace credits each slot)

refusals by rule

R-KIND 17 · R-ALC 5 · R-OPEN 2

endpoints · distinct parameters (12)

/v2/insights — feature.explainability · filter.exclude.entities · filter.exclude.tags · filter.hours · filter.location · filter.location.geohash · filter.location.radius · filter.price_level.max · filter.type · output.heatmap.boundary · signal.interests.entities · signal.interests.tags

Every Qloo request of those runs, with its parameters →