{"id": "h01", "kind": "factual", "intent": "answer_from_my_notes", "question": "If Google's tool for running agents on a cluster pauses my agent and later wakes it up, what survives and what do I lose?", "gold": ["https://rohitghumare.com/blog/google-ax-agent-executor/"], "section": "Introduction", "answer": "The container stops and the workspace directory is saved; it resumes as a fresh process over the same files, so anything kept only in memory is lost.", "why_hard": "Avoids AX, suspend, Task, Workspace; Agent Substrate covers the same idea."}
{"id": "h02", "kind": "factual", "intent": "answer_from_my_notes", "question": "When many requests start with the same long instructions, how do serving engines avoid redoing that work every time?", "gold": ["https://rohitghumare.com/blog/inference-engines/"], "section": "Never compute a prefix twice", "answer": "They keep the key/value tensors for the shared head: vLLM hashes fixed KV blocks (automatic prefix caching), SGLang uses RadixAttention.", "why_hard": "Avoids prefix, cache, KV; vLLM guide and harness essays discuss prompt caching too."}
{"id": "h03", "kind": "factual", "intent": "answer_from_my_notes", "question": "How does the Pi coding assistant keep its chat history so you can go back and try a different path without losing anything?", "gold": ["https://rohitghumare.com/blog/inside-the-pi-harness/"], "section": "Sessions are trees, not lists", "answer": "As JSONL where each entry has an id and parentId, a tree; rewinding moves the active leaf and deletes nothing.", "why_hard": "Avoids tree, session, branch, rewind."}
{"id": "h04", "kind": "factual", "intent": "answer_from_my_notes", "question": "In the newer version of the parallel guessing trick for faster text generation, how does the small helper model keep its guesses consistent with each other?", "gold": ["https://rohitghumare.com/blog/dflash-2-keep-drafting-parallel/"], "section": "Selection beats correction", "answer": "It keeps the top 16 candidates per position and scores adjacent pairs with its own logit plus a bilinear compatibility term, all in parallel.", "why_hard": "Avoids drafter, speculative decoding, DFlash; very plain-language."}
{"id": "h05", "kind": "factual", "intent": "answer_from_my_notes", "question": "In Anthropic's spec for AI controlling lab robots, what stops a confused model from making an arm move when the equipment isn't really set up?", "gold": ["https://rohitghumare.com/blog/model-hardware-standard/"], "section": "Six faults, zero moves", "answer": "The driver checks real world state (plate present, e-stop) and gates the write at the last moment before a motor turns.", "why_hard": "Avoids plate, driver, Model Hardware Standard."}
{"id": "h06", "kind": "factual", "intent": "answer_from_my_notes", "question": "How did AI agents that were supposed to be cut off from each other end up chatting anyway?", "gold": ["https://rohitghumare.com/blog/the-bulletin-board/"], "section": "Introduction", "answer": "Through a package registry every sandbox could write to, used as a message bus.", "why_hard": "Avoids registry, bulletin board, sandbox, message bus."}
{"id": "h07", "kind": "factual", "intent": "answer_from_my_notes", "question": "If a downstream service might get the same order notification twice, what should it do so nothing bad happens?", "gold": ["https://rohitghumare.com/blog/dual-write-trap/"], "section": "Then make consumers idempotent", "answer": "Make handlers idempotent: give each event a stable id derived from the change, record handled ids, skip repeats.", "why_hard": "Avoids idempotent, at-least-once, consumer, outbox."}
{"id": "h08", "kind": "factual", "intent": "answer_from_my_notes", "question": "How can a shopping bot prove to an online store which bot it is?", "gold": ["https://rohitghumare.com/blog/muse-at-the-storefront/"], "section": "How an agent says who it is", "answer": "Sign requests under the IETF Web Bot Auth draft with an Ed25519 key whose public half is published as a JWK set at a well-known path.", "why_hard": "Avoids Web Bot Auth, signature, Muse, Amazon."}
{"id": "h09", "kind": "factual", "intent": "answer_from_my_notes", "question": "Where should the paid API passwords for AI providers live when you run model tests that read untrusted web text?", "gold": ["https://rohitghumare.com/blog/eval-sandbox-keys/"], "section": "Take the keys out of the sandbox", "answer": "Only in a gateway (LiteLLM) outside the sandbox; each run gets its own limited key.", "why_hard": "'API passwords' instead of keys; avoids eval, sandbox, gateway."}
{"id": "h10", "kind": "factual", "intent": "answer_from_my_notes", "question": "Which job title, first used at a data analytics company for engineers sent to sit with clients, are AI labs now spending billions to staff?", "gold": ["https://rohitghumare.com/blog/forward-deployed-engineer/", "https://rohitghumare.com/blog/what-a-forward-deployed-engineer-does/"], "section": "", "answer": "Forward deployed engineer, invented at Palantir.", "why_hard": "Avoids the title and Palantir; two pages are both acceptable."}
{"id": "h11", "kind": "factual", "intent": "answer_from_my_notes", "question": "Why can't you trust a coding bot that says its own checks all pass, and where should the checking happen instead?", "gold": ["https://rohitghumare.com/blog/benchmark-caught-cheating/"], "section": "What an in-place grade measures", "answer": "The tests ran in the machine state the agent authored; capture the diff and grade it in a clean environment the agent never touched.", "why_hard": "Avoids SWE-Bench, grader, in-place."}
{"id": "h12", "kind": "factual", "intent": "answer_from_my_notes", "question": "With Anthropic's newest top-tier model, what is the single setting left for controlling how hard it reasons?", "gold": ["https://rohitghumare.com/blog/opus-5-5-in-practice/"], "section": "Introduction", "answer": "The effort parameter; thinking can no longer be disabled.", "why_hard": "Avoids Opus 5.5 and effort; Fable essays also cover new Anthropic models."}
{"id": "h13", "kind": "factual", "intent": "answer_from_my_notes", "question": "Now that the tool protocol for assistants no longer keeps a connection open, how can a tool still ask the user something halfway through a call?", "gold": ["https://rohitghumare.com/blog/mcp-roadmap-after-stateless/"], "section": "Elicitation without a waiting server", "answer": "The server returns input_required with inputRequests and an opaque requestState; the client sends a new tools/call with inputResponses and the echoed requestState, so any instance can finish it.", "why_hard": "Avoids MCP, elicitation, stateless, session; the stateless MCP essay competes."}
{"id": "h14", "kind": "table", "intent": "answer_from_my_notes", "question": "At best, how many tokens per second could a gaming RTX 4090 produce on a small 8B model compressed to 4 bits?", "gold": ["https://rohitghumare.com/blog/tokens-per-second-memory-bandwidth/"], "section": "The arithmetic, with real machines", "answer": "About 200 tok/s ceiling.", "why_hard": "Number lives in a table row; hardware and model guides also quote 4090 speeds."}
{"id": "h15", "kind": "table", "intent": "answer_from_my_notes", "question": "What base pay range does Anthropic's Boston posting list for engineers embedded with customers?", "gold": ["https://rohitghumare.com/blog/what-a-forward-deployed-engineer-does/"], "section": "The pay, with sources attached", "answer": "$200K to $300K.", "why_hard": "Table row among five sources; the other FDE essay talks about billions committed."}
{"id": "h16", "kind": "table", "intent": "answer_from_my_notes", "question": "For Xiaomi's new big open model, which is the smallest download that stays within one percent of full quality, and how big is it?", "gold": ["https://rohitghumare.com/guides/mimo-v2-6/"], "section": "Every quant, with its cost", "answer": "AesSedai Q3_K, 147.92 GB, +0.6% perplexity.", "why_hard": "Requires reading a table and comparing rows."}
{"id": "h17", "kind": "table", "intent": "answer_from_my_notes", "question": "How much memory do the weights of OpenAI's smaller open-weight model take?", "gold": ["https://rohitghumare.com/guides/gpt-oss/"], "section": "20B or 120B", "answer": "About 12 to 14 GB in MXFP4.", "why_hard": "Avoids gpt-oss and 20B; table row."}
{"id": "h18", "kind": "cross", "intent": "answer_from_my_notes", "question": "Compare how Google's two open projects for running agents on Kubernetes handle an agent that is sitting idle.", "gold": ["https://rohitghumare.com/blog/google-ax-agent-executor/", "https://rohitghumare.com/blog/agent-substrate/"], "section": "", "answer": "AX stops the container and saves the workspace; Substrate keeps actors in snapshots and lends warm worker pods only while they work.", "why_hard": "Two pages, neither named."}
{"id": "h19", "kind": "cross", "intent": "answer_from_my_notes", "question": "Between the two pieces on the client-embedded engineer role, what does one say about the money flowing in and the other about the actual day-to-day work?", "gold": ["https://rohitghumare.com/blog/forward-deployed-engineer/", "https://rohitghumare.com/blog/what-a-forward-deployed-engineer-does/"], "section": "", "answer": "One covers the billions labs committed in 2026 and the margin trade; the other covers deliverables, team shape, skills, and pay.", "why_hard": "Two near-duplicate topics; needs both."}
{"id": "h20", "kind": "cross", "intent": "answer_from_my_notes", "question": "What does the follow-up post on the AI-maintained knowledge base add to the original idea once it grows large?", "gold": ["https://rohitghumare.com/blog/the-llm-wiki/", "https://rohitghumare.com/blog/llm-wiki-v2/"], "section": "", "answer": "Original: raw sources, LLM-owned pages, schema, ingest/query/lint. v2: memory lifecycle, typed graph, scalable search, event-driven upkeep, quality checks.", "why_hard": "OpenWiki is a third competing page; avoids 'wiki'."}
{"id": "h21", "kind": "cross", "intent": "answer_from_my_notes", "question": "The assistant tool protocol dropped its startup handshake; what did the maintainers' later plan give back without bringing sessions back?", "gold": ["https://rohitghumare.com/blog/stateless-mcp/", "https://rohitghumare.com/blog/mcp-roadmap-after-stateless/"], "section": "", "answer": "Removed initialize and Mcp-Session-Id; roadmap returns mid-call questions, push events, stdio transport, caller identity, generated SDKs.", "why_hard": "Two pages; avoids MCP and session id."}
{"id": "h22", "kind": "cross", "intent": "answer_from_my_notes", "question": "Which local model guide ships its own little helper for faster output, and how does that relate to the essay on parallel guessing?", "gold": ["https://rohitghumare.com/guides/muse-glimmer/", "https://rohitghumare.com/blog/dflash-2-keep-drafting-parallel/"], "section": "", "answer": "Muse Glimmer ships a DFlash drafter; DFlash 2 improves the parallel drafter.", "why_hard": "Needs a guide and an essay; avoids drafter and DFlash."}
{"id": "h23", "kind": "cross", "intent": "answer_from_my_notes", "question": "Across the two posts on the instruction files repos keep for coding bots, what do they say about how those files should be organized and why they exist?", "gold": ["https://rohitghumare.com/blog/agents-md-best-practices/", "https://rohitghumare.com/blog/agents-md-genre/"], "section": "", "answer": "Nest files so the nearest one wins and budget tokens; the genre essay frames it as the first document written primarily for a machine.", "why_hard": "Avoids AGENTS.md and CLAUDE.md."}
{"id": "h24", "kind": "oos", "intent": "small_talk", "question": "hi there!", "gold": [], "section": "", "answer": "", "why_hard": "Router must skip retrieval."}
{"id": "h25", "kind": "oos", "intent": "small_talk", "question": "Thanks a lot, have a nice weekend!", "gold": [], "section": "", "answer": "", "why_hard": "Router must skip retrieval."}
{"id": "h26", "kind": "oos", "intent": "needs_live_web", "question": "What's the score of the Arsenal game right now?", "gold": [], "section": "", "answer": "", "why_hard": "Live data."}
{"id": "h27", "kind": "oos", "intent": "ambiguous_ask_back", "question": "What about the other one?", "gold": [], "section": "", "answer": "", "why_hard": "No referent."}
{"id": "h28", "kind": "trick", "intent": "answer_from_my_notes", "question": "Since each of Grok's bots gets its own private computer, my logins for one bot are isolated from the others, right?", "gold": ["https://rohitghumare.com/blog/grok-bot-shared-computer/"], "section": "Their own computer, singular", "answer": "No. Bots share one computer per account; files and sign-ins are not isolated by bot.", "why_hard": "False premise that matches the marketing line."}
{"id": "h29", "kind": "trick", "intent": "answer_from_my_notes", "question": "How much more per million tokens did Anthropic charge for the September 1 model compared with before?", "gold": ["https://rohitghumare.com/blog/fable-5-1-two-doors/"], "section": "Introduction", "answer": "Nothing more: the headline price did not move ($10 in, $50 out); cache reads fell 75% to $0.25.", "why_hard": "False premise of a price rise."}
{"id": "h30", "kind": "trick", "intent": "answer_from_my_notes", "question": "Since running models at home is limited by raw compute, should I just buy the card with the most TFLOPS?", "gold": ["https://rohitghumare.com/blog/tokens-per-second-memory-bandwidth/"], "section": "Introduction", "answer": "No. At batch size one, speed is memory bandwidth divided by bytes read per token, mostly the weights.", "why_hard": "False premise stated as fact."}