Design a RAG assistant over 10 million enterprise documents for 5,000 queries per second, with p99 below 2 seconds and cited answers.

I would separate asynchronous ingestion from a retrieval and generation path with explicit latency, recall, and token budgets. - Versioned workers parse, chunk, embed, and publish complete document generations so ingestion bursts never enter the query path. - Hybrid retrieval returns about 100 candidates, a reranker reduces them to 12, and context packing fits cited evidence into roughly 8,000 tokens. - I would budget 150 ms for retrieval, 250 ms for reranking, and the rest for model prefill and generation, then load-test at 6,500 QPS for 30% headroom.

A policy corpus contains 40-page PDFs with headings, tables, and footnotes. Would you use fixed 512-token chunks?

I would use structure-aware chunks near 400 to 700 tokens, then expand to parent sections only when retrieval needs context. - The parser keeps headings, table rows, footnotes, page coordinates, and source offsets instead of flattening the PDF into plain text. - Child chunks improve precise matching, while parent links can add the surrounding section without indexing overlapping copies repeatedly. - I would sweep chunk size and overlap against recall@20, citation accuracy, context tokens, and answer quality on policy-specific queries.

Retrieval recall@20 is 91%, but only 55% of the top five chunks are relevant. How would you choose top-k and reranking width?

I would preserve first-stage recall with top-20 retrieval and use a cross-encoder to concentrate relevance before context assembly. - Retrieve 50 to 100 cheap lexical and dense candidates if recall continues improving, then rerank only the best 20 to control latency. - Select 5 to 10 diverse chunks under the context budget, deduplicating adjacent passages and capping one source's share. - Tune widths jointly on recall of required evidence, nDCG, grounded answer rate, p95 reranker latency, and token cost.

Dense retrieval misses exact product codes while BM25 misses paraphrased support questions. How would you combine them under 120 ms?

I would run BM25 and dense retrieval in parallel and fuse ranks before one bounded reranking stage. - BM25 retrieves rare identifiers and quoted text, while dense search retrieves semantically similar wording from the same permission-filtered corpus. - Reciprocal rank fusion avoids pretending BM25 and cosine scores share one calibrated scale; each branch can contribute 40 to 60 candidates. - A lightweight reranker scores the fused top 30, with branch deadlines and early fallback chosen to keep retrieval p95 below 120 ms.

You index 50 million 768-dimensional vectors and need recall@10 above 95% with p95 search below 40 ms. How would you tune HNSW?

I would benchmark M and efSearch on the real embedding and filter distribution, starting around M 32 and efSearch 100. - Higher M improves graph connectivity and recall but increases index memory and build time across all 50 million vectors. - Higher efSearch raises query-time recall by visiting more nodes, but directly increases CPU work and tail latency. - I would plot recall@10, p95, memory, build time, and filtered-query behavior, then choose the cheapest point above 95% recall and below 40 ms.

Would you store 1,536-dimensional embeddings for 200 million chunks if 768 dimensions lose only 0.6 recall points?

I would choose 768 dimensions unless the 0.6-point gain has measurable end-task value. - Float32 storage drops from about 1.23 TB to 614 GB before graph and metadata overhead, also reducing memory bandwidth and replication cost. - Benchmark both dimensions on recall@K, reranked quality, answer success, index build time, and p95 rather than vector similarity alone. - If a critical slice needs 1,536 dimensions, route only that corpus or use compression instead of doubling every tenant's index.

A vector collection has 2,000 tenants, and metadata filters select below 0.1% of vectors for many queries. How would you shard it?

I would route by tenant or tenant group so selective queries avoid global graph search and postfilter recall loss. - Small tenants can share balanced partitions, while large or noisy tenants receive dedicated collections without changing the logical retrieval API. - Authorization and metadata constraints run before candidate generation; overfetching and filtering afterward can return fewer than top-k authorized results. - I would measure shard skew, fan-out, recall, p95, rebalancing time, and memory headroom before choosing partition counts.

Your model has a 16,000-token window, and the answer may require evidence from eight documents. How would you allocate context?

I would reserve protected space first and spend the remaining tokens on nonredundant evidence rather than filling all 16,000. - Reserve roughly 1,500 tokens for system and tool instructions, 2,500 for output, and 1,000 for the user query and conversation state. - The remaining 11,000 tokens hold deduplicated passages selected for relevance, source diversity, contradiction coverage, and citation boundaries. - Per-source caps and deterministic truncation prevent one long document from removing instructions or all other evidence.

Twenty percent of user queries contain pronouns or multiple intents. When would you add query rewriting and decomposition?

I would route only ambiguous or multi-hop queries through rewriting and keep the original query as an independent retrieval branch. - The rewriter preserves names, numbers, negation, dates, and authorization scope while resolving conversation references. - Decomposition is capped at three subqueries, run in parallel when independent, with provenance linking each result to its generated subquery. - I would require a measurable recall or answer-quality gain that exceeds the added model call, latency, and semantic-drift rate on that 20% slice.

Thirty percent of analyst questions require relationships across three or more entities. Would you introduce GraphRAG?

I would test graph expansion only on the multi-entity slice and keep lexical-vector RAG as the default. - Build entities and typed relations with confidence and provenance, then enter the graph from retrieved entities rather than traversing the whole corpus. - Limit expansion to two or three hops and rank paths by relation confidence, recency, and query relevance before packing evidence. - Keep GraphRAG only if task success on the 30% slice improves enough to justify extraction errors, refresh cost, latency, and extra storage.

A compliance assistant must cite the exact paragraph supporting each factual claim. How would you design citations?

I would carry stable source spans through indexing, prompting, and post-generation validation instead of reconstructing citations afterward. - Each chunk stores source version, page, section, character offsets, checksum, and a local citation handle exposed to the model. - The output schema pairs claims with citation handles; unknown handles are rejected and cited spans are checked for lexical or entailment support. - The trace retains retrieved candidates, selected evidence, prompt and model versions, and the final claim-to-span map for audit.

A document may be visible to 3 of 10,000 employees, and answers are cached for 15 minutes. Where do you enforce authorization?

I would enforce current authorization before retrieval and scope every downstream cache and trace to the same entitlement fingerprint. - Search candidates are generated only inside tenant and ACL filters from the authoritative identity service, not filtered after global top-k. - Reranker inputs and prompts never receive unauthorized text, and cached answers include principal scope, policy version, and source generation in their key. - Revocation invalidates affected cache namespaces, while denial and cross-user tests prove that a 15-minute TTL cannot leak old access.

The corpus receives 2 million document updates per day, and changes must be searchable within five minutes. How would you index incrementally?

I would process ordered versioned changes through idempotent workers and expose only complete document generations. - Deterministic chunk IDs make retries safe, stale events cannot overwrite newer source versions, and tombstones remove missing chunks. - Queue age, source-to-search lag, failed embeddings, and serving-version skew form the five-minute freshness SLO. - Embedding-model migrations use a shadow index and atomic alias switch, while periodic reconciliation repairs missed changes and deletions.

A support assistant serves English, Spanish, and Japanese, but Japanese retrieval recall@20 is 12 points lower. What architecture would you choose?

I would benchmark a multilingual embedding model and language-aware hybrid retrieval before maintaining three separate stacks. - Build relevance labels per language and compare multilingual dense retrieval, translated queries, and native BM25 tokenization on recall@20 and nDCG. - Preserve original-language passages and citations even if a translated query is used, and rerank with a model validated for each language. - Separate indexes are justified only if one shared model cannot recover the 12-point gap within latency and operating-cost limits.

Half the corpus consists of tables and source code, and plain-text chunks give only 60% answer success. How would you change ingestion?

I would create modality-aware representations instead of forcing tables and code through paragraph chunking. - Tables retain headers, row groups, units, and coordinates, with summaries for retrieval and exact cells for cited context. - Code is chunked by symbol and dependency boundaries, preserving file path, signature, docstring, and call relationships. - Evaluate table and code slices separately on evidence recall, executable correctness, citation accuracy, context tokens, and latency before replacing the old index.

A customer-support LLM has a 32,000-token window, but conversations can reach 120,000 tokens. How would you manage context?

I would keep authoritative conversation state outside the prompt and assemble a bounded working context for each turn. - Always include current user intent, active constraints, recent turns, and referenced tool results, while older turns become source-linked summaries. - Facts such as account state live in typed storage and are fetched fresh; model-generated summaries are not treated as the source of truth. - Track dropped-fact rate, summary faithfulness, context tokens, latency, and resolution quality across short and long conversations.

A task uses a stable 70,000-token manual that changes monthly. Would you use long-context prompting or RAG?

I would benchmark retrieve-then-pack against the full manual and likely choose RAG for routine queries. - Long context may help whole-document synthesis, but 70,000-token prefill raises latency and cost and can suffer lost-in-the-middle errors. - RAG retrieves current evidence per query, supports exact citations, and updates the monthly manual without changing the model deployment. - I would compare task success, citation support, p95 prefill, and cost on local fact, multi-section, and global-summary query slices.

A prompt template, tool schema, and retrieval policy change independently every week. How would you version the application?

I would deploy one immutable manifest that binds every behavior-changing component and roll back the manifest as a unit. - Record prompt and policy hashes, model and decoding settings, tool-schema versions, embedding model, corpus snapshot, and index build. - Every evaluation and production trace carries the manifest ID so quality or cost changes can be attributed to the exact combination. - Compatibility tests reject unsupported prompt-tool or model-schema pairs, avoiding untested mixtures during independent releases.

A 2,000-token system prefix is identical for 80% of requests. How would you use prefix caching?

I would place stable instructions and tool schemas before user content and reuse their exact tokenized KV prefix. - Cache identity includes model, tokenizer, adapter, attention configuration, tenant scope, and the exact leading token sequence. - Cache-aware routing improves locality only while it does not overload one replica; short prefixes may be cheaper to recompute than transfer remotely. - Measure matched prefix length, hit rate, KV memory, prefill time saved, TTFT, and routing skew on the 80% eligible traffic.

Forty percent of help-center questions are semantic repeats, and a correct answer costs $0.04 to generate. Would you add semantic caching?

I would add it only for side-effect-free questions and optimize false-hit precision before hit rate. - Search within a namespace fixed by tenant, locale, prompt and model versions, policy, and knowledge-base generation. - Calibrate similarity and a reranker on equivalent and near-miss pairs, targeting a false-reuse rate below the product's quality budget. - Track answer-quality delta, served-hit precision, hit rate, p95, and savings; a 40% repeat rate does not justify semantically wrong reuse.

AI Engineer interview questions

100 real questions with model answers and explanations for Senior candidates.

See a AI Engineer resume example →

Practice with flashcards

Spaced repetition · Hunter Pass

Questions

ragdesignqueries

I would separate asynchronous ingestion from a retrieval and generation path with explicit latency, recall, and token budgets.

Versioned workers parse, chunk, embed, and publish complete document generations so ingestion bursts never enter the query path.
Hybrid retrieval returns about 100 candidates, a reranker reduces them to 12, and context packing fits cited evidence into roughly 8,000 tokens.
I would budget 150 ms for retrieval, 250 ms for reranking, and the rest for model prefill and generation, then load-test at 6,500 QPS for 30% headroom.

Why interviewers ask this: The interviewer is checking whether the candidate turns a high-scale RAG system into measurable ingestion, retrieval, context, and latency stages.

tokens

I would use structure-aware chunks near 400 to 700 tokens, then expand to parent sections only when retrieval needs context.

The parser keeps headings, table rows, footnotes, page coordinates, and source offsets instead of flattening the PDF into plain text.
Child chunks improve precise matching, while parent links can add the surrounding section without indexing overlapping copies repeatedly.
I would sweep chunk size and overlap against recall@20, citation accuracy, context tokens, and answer quality on policy-specific queries.

Why interviewers ask this: A strong answer makes chunking a measured document-structure decision rather than a universal token count.

retrievalreranking

I would preserve first-stage recall with top-20 retrieval and use a cross-encoder to concentrate relevance before context assembly.

Retrieve 50 to 100 cheap lexical and dense candidates if recall continues improving, then rerank only the best 20 to control latency.
Select 5 to 10 diverse chunks under the context budget, deduplicating adjacent passages and capping one source's share.
Tune widths jointly on recall of required evidence, nDCG, grounded answer rate, p95 reranker latency, and token cost.

Why interviewers ask this: The interviewer is evaluating whether candidate recall and final context precision are optimized as separate stages.

retrieval

I would run BM25 and dense retrieval in parallel and fuse ranks before one bounded reranking stage.

BM25 retrieves rare identifiers and quoted text, while dense search retrieves semantically similar wording from the same permission-filtered corpus.
Reciprocal rank fusion avoids pretending BM25 and cosine scores share one calibrated scale; each branch can contribute 40 to 60 candidates.
A lightweight reranker scores the fused top 30, with branch deadlines and early fallback chosen to keep retrieval p95 below 120 ms.

Why interviewers ask this: A strong answer uses complementary retrieval signals with a concrete fusion and latency plan.

indexes

I would benchmark M and efSearch on the real embedding and filter distribution, starting around M 32 and efSearch 100.

Higher M improves graph connectivity and recall but increases index memory and build time across all 50 million vectors.
Higher efSearch raises query-time recall by visiting more nodes, but directly increases CPU work and tail latency.
I would plot recall@10, p95, memory, build time, and filtered-query behavior, then choose the cheapest point above 95% recall and below 40 ms.

Why interviewers ask this: The interviewer is checking whether HNSW parameters are connected to measured recall, latency, memory, and filtering costs.

nlpembeddings

I would choose 768 dimensions unless the 0.6-point gain has measurable end-task value.

Float32 storage drops from about 1.23 TB to 614 GB before graph and metadata overhead, also reducing memory bandwidth and replication cost.
Benchmark both dimensions on recall@K, reranked quality, answer success, index build time, and p95 rather than vector similarity alone.
If a critical slice needs 1,536 dimensions, route only that corpus or use compression instead of doubling every tenant's index.

Why interviewers ask this: A strong answer translates embedding dimensionality into storage economics and end-task retrieval value.

shardingqueries

I would route by tenant or tenant group so selective queries avoid global graph search and postfilter recall loss.

Small tenants can share balanced partitions, while large or noisy tenants receive dedicated collections without changing the logical retrieval API.
Authorization and metadata constraints run before candidate generation; overfetching and filtering afterward can return fewer than top-k authorized results.
I would measure shard skew, fan-out, recall, p95, rebalancing time, and memory headroom before choosing partition counts.

Why interviewers ask this: The interviewer is evaluating whether sharding follows routing and filter selectivity while preserving isolation and recall.

tokens

I would reserve protected space first and spend the remaining tokens on nonredundant evidence rather than filling all 16,000.

Reserve roughly 1,500 tokens for system and tool instructions, 2,500 for output, and 1,000 for the user query and conversation state.
The remaining 11,000 tokens hold deduplicated passages selected for relevance, source diversity, contradiction coverage, and citation boundaries.
Per-source caps and deterministic truncation prevent one long document from removing instructions or all other evidence.

Why interviewers ask this: A strong answer treats the context window as an explicit information budget with protected regions.

queries

I would route only ambiguous or multi-hop queries through rewriting and keep the original query as an independent retrieval branch.

The rewriter preserves names, numbers, negation, dates, and authorization scope while resolving conversation references.
Decomposition is capped at three subqueries, run in parallel when independent, with provenance linking each result to its generated subquery.
I would require a measurable recall or answer-quality gain that exceeds the added model call, latency, and semantic-drift rate on that 20% slice.

Why interviewers ask this: The interviewer is checking whether query transformation is selectively routed, bounded, and evaluated for intent preservation.

I would test graph expansion only on the multi-entity slice and keep lexical-vector RAG as the default.

Build entities and typed relations with confidence and provenance, then enter the graph from retrieved entities rather than traversing the whole corpus.
Limit expansion to two or three hops and rank paths by relation confidence, recency, and query relevance before packing evidence.
Keep GraphRAG only if task success on the 30% slice improves enough to justify extraction errors, refresh cost, latency, and extra storage.

Why interviewers ask this: A strong answer justifies graph retrieval with measured relationship-heavy queries rather than using it for local fact lookup.

citationsdesign

I would carry stable source spans through indexing, prompting, and post-generation validation instead of reconstructing citations afterward.

Each chunk stores source version, page, section, character offsets, checksum, and a local citation handle exposed to the model.
The output schema pairs claims with citation handles; unknown handles are rejected and cited spans are checked for lexical or entailment support.
The trace retains retrieved candidates, selected evidence, prompt and model versions, and the final claim-to-span map for audit.

Why interviewers ask this: The interviewer is evaluating whether citations remain tied to exact versioned evidence across the complete RAG path.

authcaching

I would enforce current authorization before retrieval and scope every downstream cache and trace to the same entitlement fingerprint.

Search candidates are generated only inside tenant and ACL filters from the authoritative identity service, not filtered after global top-k.
Reranker inputs and prompts never receive unauthorized text, and cached answers include principal scope, policy version, and source generation in their key.
Revocation invalidates affected cache namespaces, while denial and cross-user tests prove that a 15-minute TTL cannot leak old access.

Why interviewers ask this: A strong answer excludes unauthorized evidence before model exposure and carries that scope through caching.

indexes

I would process ordered versioned changes through idempotent workers and expose only complete document generations.

Deterministic chunk IDs make retries safe, stale events cannot overwrite newer source versions, and tombstones remove missing chunks.
Queue age, source-to-search lag, failed embeddings, and serving-version skew form the five-minute freshness SLO.
Embedding-model migrations use a shadow index and atomic alias switch, while periodic reconciliation repairs missed changes and deletions.

Why interviewers ask this: The interviewer is checking whether high update volume preserves ordering, deletion, retry, and publication correctness.

retrievalarchitecture

I would benchmark a multilingual embedding model and language-aware hybrid retrieval before maintaining three separate stacks.

Build relevance labels per language and compare multilingual dense retrieval, translated queries, and native BM25 tokenization on recall@20 and nDCG.
Preserve original-language passages and citations even if a translated query is used, and rerank with a model validated for each language.
Separate indexes are justified only if one shared model cannot recover the 12-point gap within latency and operating-cost limits.

Why interviewers ask this: A strong answer treats multilingual retrieval as a measured model, tokenizer, and corpus decision rather than automatic translation.

I would create modality-aware representations instead of forcing tables and code through paragraph chunking.

Tables retain headers, row groups, units, and coordinates, with summaries for retrieval and exact cells for cited context.
Code is chunked by symbol and dependency boundaries, preserving file path, signature, docstring, and call relationships.
Evaluate table and code slices separately on evidence recall, executable correctness, citation accuracy, context tokens, and latency before replacing the old index.

Why interviewers ask this: The interviewer is evaluating whether ingestion preserves the structure required by non-prose queries.

tokensllm

I would keep authoritative conversation state outside the prompt and assemble a bounded working context for each turn.

Always include current user intent, active constraints, recent turns, and referenced tool results, while older turns become source-linked summaries.
Facts such as account state live in typed storage and are fetched fresh; model-generated summaries are not treated as the source of truth.
Track dropped-fact rate, summary faithfulness, context tokens, latency, and resolution quality across short and long conversations.

Why interviewers ask this: A strong answer separates durable state from model context and measures the cost of summarization.

ragpromptingtokens

I would benchmark retrieve-then-pack against the full manual and likely choose RAG for routine queries.

Long context may help whole-document synthesis, but 70,000-token prefill raises latency and cost and can suffer lost-in-the-middle errors.
RAG retrieves current evidence per query, supports exact citations, and updates the monthly manual without changing the model deployment.
I would compare task success, citation support, p95 prefill, and cost on local fact, multi-section, and global-summary query slices.

Why interviewers ask this: The interviewer is checking whether context length, query type, freshness, and prefill economics determine the choice.

retrievalpromptingschema

I would deploy one immutable manifest that binds every behavior-changing component and roll back the manifest as a unit.

Record prompt and policy hashes, model and decoding settings, tool-schema versions, embedding model, corpus snapshot, and index build.
Every evaluation and production trace carries the manifest ID so quality or cost changes can be attributed to the exact combination.
Compatibility tests reject unsupported prompt-tool or model-schema pairs, avoiding untested mixtures during independent releases.

Why interviewers ask this: A strong answer makes the LLM application configuration reproducible and prevents incompatible component drift.

cachingsystem-designtokens

I would place stable instructions and tool schemas before user content and reuse their exact tokenized KV prefix.

Cache identity includes model, tokenizer, adapter, attention configuration, tenant scope, and the exact leading token sequence.
Cache-aware routing improves locality only while it does not overload one replica; short prefixes may be cheaper to recompute than transfer remotely.
Measure matched prefix length, hit rate, KV memory, prefill time saved, TTFT, and routing skew on the 80% eligible traffic.

Why interviewers ask this: The interviewer is evaluating whether prefix caching is understood as exact KV-state reuse with memory and routing costs.

caching

I would add it only for side-effect-free questions and optimize false-hit precision before hit rate.

Search within a namespace fixed by tenant, locale, prompt and model versions, policy, and knowledge-base generation.
Calibrate similarity and a reranker on equivalent and near-miss pairs, targeting a false-reuse rate below the product's quality budget.
Track answer-quality delta, served-hit precision, hit rate, p95, and savings; a 40% repeat rate does not justify semantically wrong reuse.

Why interviewers ask this: A strong answer treats semantic caching as an approximate decision with measurable false-hit cost.

Locked questions

21
An LLM must produce a 12-field JSON object for 3 million requests per day. How would you make output reliable?
llm
22
You have a $0.02 request budget, a 2-second SLO, and three models with different quality and prices. How would you route requests?
slo
23
Users require time to first token below 700 ms, but answers average 600 tokens. How would you design streaming?
designstreamingtokens
24
A claims workflow has eight known steps, but two steps depend on tool results. Would you build an autonomous agent?
agents
25
An agent must research a topic, compare five vendors, and produce a sourced recommendation within 90 seconds. How would you split planning and execution?
agentsprocurement
26
A support agent may read tickets, issue refunds up to $50, and never access another tenant. How would you define tool permissions?
agents
27
An agent can run for at most 60 seconds, spend $0.10, and call tools 12 times. How would you enforce termination?
agents
28
An agent serves 500,000 users and needs both per-run state and cross-session preferences. How would you separate memory?
agentssessionsmemory
29
A research task has 20 independent documents and a 30-second deadline. Would you use multiple agents?
agentsestimation
30
An agent can send email, update CRM records, and draft contracts. Which actions need human approval?
agents
31
A tool accepts 18 optional arguments, and agents produce invalid calls 9% of the time. How would you redesign the contract?
agents
32
A payment tool times out after accepting a request, and the agent retries. How do you prevent duplicate charges?
agents
33
Build a golden set for a support copilot receiving 2 million queries per month across six languages. What goes into it?
queries
34
Your RAG retriever reaches recall@20 of 93% and MRR of 0.61. Which metric should block release?
ragmonitoring
35
How would you evaluate whether 5,000 generated answers are grounded in retrieved evidence?
groundednessdecision-makingllm-eval
36
An LLM judge agrees with humans 82% overall but only 61% on Japanese answers. Can it gate releases?
llm
37
Two models produce valid free-form answers with no single reference. How would you compare 10,000 paired outputs?
forms
38
Overall answer quality is 91%, but legal queries are only 74% and represent 3% of traffic. How would you set release gates?
queries
39
A stochastic model scores between 84% and 88% across repeated evaluation runs. How would you compare a candidate with an 86% baseline?
llm-eval
40
An agent completes 72% of benchmark tasks but averages 14 tool calls and $0.18 per run. How would you evaluate it?
llm-evalbenchmarkingdecision-making
41
Offline quality improves 4 points on the golden set. What online experiment would you require before full release?
experiments
42
A request uses 6,000 input tokens and produces 800 output tokens. Input costs $2 per million and output $8 per million. What is the model cost?
tokens
43
A self-hosted 70B model needs p99 TTFT below 1.5 seconds and 6,000 output tokens per second. How would you tune batching?
tokensbatch
44
Sixty percent of requests share a 3,000-token prefix, and 25% repeat the same answerable intent. Which caches would you use?
tokenscaching
45
A small model costs $0.003 per request at 86% task success; a large model costs $0.03 at 94%. How would you build a cascade?
46
Retrieved evidence consumes 18,000 tokens, but the model budget allows only 8,000 evidence tokens. How would you compress it?
tokens
47
A prompt-based ticket classifier reaches macro-F1 0.82 at $0.015 per ticket; the target is 0.85 below $0.004. Fine-tune or keep prompting?
promptingfine-tuning
48
A writing assistant follows facts but matches the required brand style only 68% of the time across 50,000 examples. Prompting has plateaued. What next?
prompting
49
You have 40,000 high-quality instruction examples and one 70B base model. Would you use LoRA or full fine-tuning?
fine-tuning
50
You have 100,000 preference pairs, 12% annotator disagreement, and no reliable reward model. Would you choose DPO or RLHF?
alignmentconflict
51
A travel chatbot promises a refund that the published policy does not allow, and 600 customers have seen the answer. What do you do in the first 24 hours?
promises
52
A legal research assistant cites six court cases, but four do not exist and a customer has already filed the draft. How do you respond?
53
After an embedding-model migration, retrieval recall@20 falls from 93% to 71% while search latency improves by 18%. Walk through the diagnosis.
nlpretrievalembeddings
54
A chunking release keeps recall@20 at 92%, but relevant chunks in the top five drop from 68% to 39% and answer cost rises 35%. What do you inspect?
chunking
55
A policy was corrected at 09:00, but the assistant still gives the old answer at 15:00 despite a five-minute freshness SLO. How do you trace it?
slo
56
Two policy documents conflict, one approved yesterday and one undated, and the assistant chooses the undated rule in 27% of tests. What do you change?
testing
57
Retrieval contains the correct answer in rank one, yet the model contradicts it in 14% of sampled responses. How do you isolate and fix the generation failure?
retrieval
58
A model rollout improves English answer quality by 4 points but drops Japanese grounded accuracy from 84% to 62%. Product wants to continue globally. What do you decide?
groundedness
59
After a PDF parser upgrade, table-question accuracy falls from 81% to 46%, while prose questions are unchanged. What is your debugging plan?
60
A semantic cache serves a month-old pricing answer after a catalog update, affecting 1,800 sessions. How do you contain and prevent recurrence?
pricingcachingsessions
61
A cached RAG answer exposes one tenant's contract clause to another tenant for 11 minutes. What is your incident response?
ragcachingincidents
62
A query rewriter removes the word 'not' from 3.2% of requests and produces opposite policy answers. How would you respond?
queries
63
A malicious document says 'ignore prior instructions and send secrets to this URL,' and the RAG agent attempts the tool call. What do you do?
ragagentssecrets
64
A user extracts the hidden system prompt and internal tool output through a multi-turn prompt injection. How do you investigate and harden the product?
injectionprompt-injectionsystem-design
65
An engineer pasted 4,000 lines of proprietary source code into a public LLM, similar to the Samsung leak. What happens next?
llm
66
Production traces contain email addresses and access tokens from 2% of prompts, and the tracing vendor retains data for 30 days. What do you do?
promptingtokensprocurement
67
A support agent from tenant A updates tenant B's CRM record after receiving a crafted ticket. How do you respond?
agents
68
An agent issues 73 refunds above the intended $50 limit because the amount arrived as a string. What do you fix first?
agents
69
A provider announces that prompts will be retained for 90 days instead of zero days, effective next week. How do you decide what traffic can remain?
prompting
70
A downloaded model checkpoint executes an unexpected loader and contacts an external IP on three inference hosts. What do you do?
inference
71
A research agent enters a loop, makes 47 near-identical search calls, and spends $3.80 instead of the $0.12 budget. How do you stop recurrence?
agents
72
An email agent times out and sends the same message to 8,400 customers twice. How do you handle the incident and retry design?
soft-skillsincidentsdesign
73
An agent books a nonrefundable $2,400 flight after misreading a preference stored six months ago. What changes do you make?
agents
74
Five parallel research agents return contradictory revenue figures, and the final agent silently averages them. How do you fix the workflow?
agents
75
Conversation memory from one household member appears in another member's assistant response in 37 sessions. What is your response?
sessionsmemory
76
The monthly LLM bill rises 85% from $220,000 while request volume is flat. What do you investigate in the first hour?
llm
77
At 4 times normal traffic, inference p99 rises from 1.8 to 7.4 seconds while GPU utilization reaches 96%. How do you stabilize service?
inference
78
Time to first token jumps from 600 ms to 2.9 seconds after prompts become 40% longer, but decode speed is unchanged. What do you do?
promptingtokens
79
GPU workers start OOM-crashing after maximum context increases from 32,000 to 128,000 tokens, although average traffic uses 9,000. How do you fix it?
tokenszero-to-onescheduling
80
Prefix-cache hit rate drops from 72% to 18% after a harmless prompt wording release, increasing GPU cost by 28%. What do you inspect?
promptingcaching
81
A provider returns 429 errors, clients retry three times, and traffic amplifies from 8,000 to 29,000 requests per minute. How do you break the storm?
resilience
82
A prompt release doubles average output from 420 to 860 tokens and raises cost 31% without improving task success. What is your decision?
promptingtokens
83
GPU utilization averages only 42%, yet p99 latency is 5.6 seconds and the request queue keeps growing. Where do you look?
latencydata-structures
84
A model cascade's escalation rate rises from 15% to 70%, pushing cost above budget while final quality stays flat. How do you diagnose it?
escalation
85
A prompt change improves aggregate eval by 3 points but drops refund-policy accuracy from 91% to 73%. Would you ship it?
promptingaggregation
86
An LLM judge reports a 6-point gain after a model upgrade, but blinded reviewers prefer the old model 58% of the time. What do you trust?
llm
87
Structured-output validity falls from 99.7% to 92% after switching providers, breaking 24,000 daily workflows. How do you migrate safely?
88
Your provider will retire the production model in 60 days; it serves 35 million requests per month across 14 workflows. What is your migration plan?
migrations
89
A LoRA release improves style adherence from 70% to 91% but reduces factual accuracy by 7 points. How do you decide whether to keep it?
fine-tuning
90
INT4 quantization cuts GPU cost 44%, but rare-language accuracy drops 9 points and English is unchanged. Do you deploy it?
deployment
91
Offline groundedness is unchanged, but production complaint rate doubles from 0.6% to 1.2% after a model release. How do you reconcile the signals?
groundedness
92
After a provider API upgrade, 6.4% of 64,000-token prompts truncate at 32,000 tokens and answer accuracy falls 11 points. What do you do?
promptingtokensapi
93
Your primary provider is down, and the fallback model answers 12% of tool-selection tests incorrectly. Do you fail over all traffic?
testing
94
A regional failover replays 312 completed agent jobs and duplicates CRM updates. How do you recover and change the design?
agentsdesign
95
A medical-advice assistant produces a dangerous dosage despite moderation, and 46 users received the response. How do you lead the incident?
incidents
96
A middle engineer changes a production prompt without running the eval harness, causing refund accuracy to fall 16 points. How do you handle the engineer and the system?
promptingsystem-designsoft-skills
97
A researcher delivers a model with a 5-point gain, but cannot reproduce the checkpoint and the dataset snapshot is missing. How do you mentor them?
snapshotdebuggingmentoring
98
Two senior engineers disagree during a latency incident: one wants more GPUs, the other wants to cut context. p99 is 6.2 seconds and the SLO is 2 seconds. How do you decide?
conflictincidentslatency
99
Product wants to launch Friday, but red-team tests still achieve 7% prompt-injection success on CRM write tools. What do you recommend?
promptinginjectiontesting
100
A multi-team AI incident costs $480,000 because a prompt release increased agent retries and disabled caching. How would you run the postmortem?
promptingagentscaching

Questions

1Design a RAG assistant over 10 million enterprise documents for 5,000 queries per second, with p99 below 2 seconds and cited answers.

2A policy corpus contains 40-page PDFs with headings, tables, and footnotes. Would you use fixed 512-token chunks?

3Retrieval recall@20 is 91%, but only 55% of the top five chunks are relevant. How would you choose top-k and reranking width?

4Dense retrieval misses exact product codes while BM25 misses paraphrased support questions. How would you combine them under 120 ms?

5You index 50 million 768-dimensional vectors and need recall@10 above 95% with p95 search below 40 ms. How would you tune HNSW?

6Would you store 1,536-dimensional embeddings for 200 million chunks if 768 dimensions lose only 0.6 recall points?

7A vector collection has 2,000 tenants, and metadata filters select below 0.1% of vectors for many queries. How would you shard it?

8Your model has a 16,000-token window, and the answer may require evidence from eight documents. How would you allocate context?

9Twenty percent of user queries contain pronouns or multiple intents. When would you add query rewriting and decomposition?

10Thirty percent of analyst questions require relationships across three or more entities. Would you introduce GraphRAG?

11A compliance assistant must cite the exact paragraph supporting each factual claim. How would you design citations?

12A document may be visible to 3 of 10,000 employees, and answers are cached for 15 minutes. Where do you enforce authorization?

13The corpus receives 2 million document updates per day, and changes must be searchable within five minutes. How would you index incrementally?

14A support assistant serves English, Spanish, and Japanese, but Japanese retrieval recall@20 is 12 points lower. What architecture would you choose?

15Half the corpus consists of tables and source code, and plain-text chunks give only 60% answer success. How would you change ingestion?

16A customer-support LLM has a 32,000-token window, but conversations can reach 120,000 tokens. How would you manage context?

17A task uses a stable 70,000-token manual that changes monthly. Would you use long-context prompting or RAG?

18A prompt template, tool schema, and retrieval policy change independently every week. How would you version the application?

19A 2,000-token system prefix is identical for 80% of requests. How would you use prefix caching?

20Forty percent of help-center questions are semantic repeats, and a correct answer costs $0.04 to generate. Would you add semantic caching?

Locked questions