A generic AI copilot can be copied by competitors within one quarter. How would you choose a product wedge that can become a durable moat?

I would start with one high-friction workflow where we can own the outcome, not with a general chat surface. - I would target the approval step that currently takes operations teams 45 minutes, because embedding into that workflow creates switching cost beyond model quality. - I would secure permission to learn from accepted edits and reviewer reasons, building proprietary task data that competitors cannot buy from a model vendor. - I would pair the workflow with distribution through the customer's existing CRM or help desk, then test whether 30% of weekly users complete the full job inside our product.

A support team resolves tickets in 14 minutes using templates and search. How would you set the non-AI baseline and the automation ceiling before funding an AI assistant?

I would benchmark AI against the best realistic rules-and-search flow and cap automation by the cases the product can safely own. - I would measure handle time, first-contact resolution, and correction rate for templates plus improved search before attributing any gain to AI. - I would label the share of tickets that require policy judgment or account actions; if 35% need human authority, 65% is the initial automation ceiling. - I would fund the assistant only if it beats the baseline on resolved cost per ticket while keeping factual corrections below the agreed 3% limit.

You must choose between an AI meeting summarizer and an AI contract reviewer for the next quarter. How would you apply a value, risk, and feasibility score?

I would use the score to expose assumptions, but normalize value to the same time horizon before comparing the options. - I would annualize both benefits: eight saved hours per month at a $75 loaded hourly cost is $7,200 per user-year, while contract review may avoid an expected $20,000 annual loss. - I would score risk by severity and reversibility; a missed meeting detail is correctable, while a missed contract clause can create legal loss. - I would score feasibility from a 100-case eval, required integrations, and review cost, then compare risk-adjusted annual value with explicit confidence ranges.

Leadership has five AI ideas but only two squads for six months. How would you frame a portfolio hypothesis instead of ranking isolated features?

I would define a small set of portfolio beliefs that can be tested by several related bets. - One hypothesis might be that reducing analyst preparation time by 50% increases weekly completed reports, covering extraction, drafting, and review features. - I would reserve roughly 70% of capacity for the strongest workflow, 20% for an adjacent test, and 10% for cheap probes of model capabilities. - Each bet would have a shared user outcome, a model-quality gate, and a stop date so one impressive demo cannot consume the whole portfolio.

A document classifier needs proprietary labels, but only current power users generate feedback. What data acquisition and flywheel strategy would you choose?

I would acquire data through the core workflow while preventing today's power-user behavior from defining every future user. - I would ask reviewers to confirm or correct only decisions that affect their work, capturing the original input, final label, and reason with explicit usage consent. - I would seed underrepresented industries through paid design partners and targeted annotation instead of waiting for organic volume. - I would monitor label share and error rate by customer size, language, and workflow so the flywheel does not reinforce the preferences of the loudest 10% of users.

Your team needs 20,000 labeled support conversations across six issue types. How would you design the annotation operating model?

I would use a tiered annotation model with clear ownership for guidelines, quality, and difficult cases. - Trained vendors can label routine intent and evidence spans, while internal support specialists adjudicate policy-sensitive or ambiguous examples. - I would pilot 300 cases, refine examples until pairwise agreement exceeds the agreed target such as Cohen's kappa of 0.75, then scale volume. - Weekly audits would sample each annotator and class, with guideline versions attached to labels so changes remain traceable.

Experts cost $12 per labeled case, and the model sees one million unlabeled cases monthly. When does active learning make product-economic sense?

I would use active learning only if selecting informative cases reduces total labeling cost without distorting the production mix. - I would compare a random-labeling curve with uncertainty or disagreement sampling, measuring quality gained per 1,000 expert dollars. - I would keep a random 20% sample because uncertainty sampling can overfocus on noisy edge cases and hide common-case regressions. - I would continue if the same release threshold is reached with materially fewer labels, for example 4,000 instead of 10,000, after including selection and review overhead.

The team proposes generating 80% of a new eval set synthetically because real examples are scarce. What role would you allow synthetic data to play?

I would use synthetic data to expand known scenarios, but never let it define the release truth by itself. - Synthetic cases are useful for rare formats, adversarial variations, and controlled difficulty where each expected behavior can be reviewed. - I would keep a separate human-sourced set from real workflows because generated examples inherit the generator's style, blind spots, and policy assumptions. - Release decisions would report real and synthetic slices separately, with at least a few hundred representative real cases before broad launch.

An AI drafting feature scores 88% overall. How would you segment the eval before deciding whether that number is acceptable?

I would segment by who uses the feature, what job they are doing, and the consequence of a wrong output. - User slices would include new versus expert users, enterprise versus self-serve accounts, and every supported language with meaningful traffic. - Job slices would separate short rewrites, multi-source drafts, and policy-sensitive responses because one average hides different difficulty. - Risk slices would have independent gates, such as 95% factual pass for regulated text even if low-stakes brainstorming can ship at 85% usefulness.

Model A has 92% quality, $0.18 cost, and 4.5-second latency; model B has 88% quality, $0.04 cost, and 1.4-second latency. How would you define a product utility function?

I would express each option as expected value per successful task so percentages, dollars, and seconds are not added as if they shared a unit. - Quality would have a hard floor, such as no model below 87% task success, so low cost cannot compensate for unusable output. - Above that floor, I would estimate contribution from accepted completions, then subtract inference and review cost plus the monetized completion loss caused by latency. - I would validate those estimates in an online test and publish sensitivity ranges, because routing should not hinge on one guessed dollar value for a quality point.

A classifier routes billing, cancellation, fraud, and general support tickets. How would you set per-class thresholds from business loss?

I would set each threshold from the cost of its false positive and false negative rather than use 0.5 for every class. - Missing fraud may cost $800 per case, so I would accept more false alerts if review capacity and customer friction remain within limits. - A false cancellation route may trigger an unwanted retention flow, so that class needs a higher precision threshold than general support. - I would calculate expected loss on a representative validation set and revisit thresholds when case value, prevalence, or reviewer cost changes.

An AI reviewer can auto-decide 70% of cases at 96% precision, while humans can review only 2,000 cases per day. How would you choose confidence and coverage targets?

I would choose the highest safe coverage that keeps the uncertain queue within real human capacity. - I would plot precision against coverage by risk tier and set auto-decision thresholds separately for low-loss and high-loss cases. - At forecast volume, the abstained share must stay below 2,000 daily reviews with a buffer for peaks, vacations, and appeals. - If demand exceeds capacity, I would narrow eligible cases or improve prioritization rather than silently lower confidence and overload reviewers.

A marketplace ranking model improves nDCG by 6% offline but may concentrate demand among top sellers. What would you require before launch?

I would treat nDCG as a retrieval signal and require evidence that the marketplace outcome improves without damaging supply health. - The online primary metric could be completed purchases per search, with guardrails for returns, buyer cancellations, and time to first sale for new sellers. - I would segment seller exposure and concentration, because a conversion lift can come from starving the long tail rather than better matching. - A switchback or market-level test would reduce interference between treatment and control when the same inventory serves both groups.

How would you set up a human evaluation program for an enterprise answer assistant used across finance, HR, and operations?

I would build a blinded, recurring program with domain-specific rubrics and an adjudication path. - Reviewers would score factual support, task completion, and harmfulness on anchored scales, with examples of what a 1, 3, and 5 mean. - Each release sample would include paired baseline and candidate answers, balanced across domains, languages, and difficult production cases. - I would track agreement, reviewer drift, and cost per evaluated case, then send disagreements to a domain owner rather than average them away.

The team wants an LLM judge to replace most human review. What limits would you set from a product-management perspective?

I would use an LLM judge for scale only after proving where it agrees with humans and where it does not. - I would calibrate it on blinded domain-reviewer labels and report agreement by language, answer length, and risk class, not one overall correlation. - Humans would retain ownership of policy interpretation, high-impact failures, and a random audit sample because judges can favor style, position, and familiar wording. - A judge-score change would block release only with stable prompts, pinned versions, repeated runs, and a human appeal path for borderline cases.

Who should own eval-as-CI when model, prompt, data, and product teams all contribute to an AI feature?

I would give one product-engineering pair accountable ownership while distributing specific gates to domain owners. - Product owns the user-risk taxonomy and release thresholds; engineering owns deterministic execution, version capture, and blocking behavior in CI. - Applied ML owns model and prompt regressions, while legal or Trust and Safety approves the policy slices they are qualified to judge. - The RFC would name who updates cases, who may waive a failed gate, and how every waiver expires, preventing a shared system from becoming ownerless.

Offline task success rises from 82% to 89%, but what bridge would you require before predicting an online user outcome?

I would state the causal bridge explicitly and test each link rather than claim that seven offline points equal product value. - The hypothesis might be that better factual completion reduces edits, which cuts time to send and increases weekly completed tasks. - I would verify the first link in logged or moderated sessions, measuring edit distance and completion time on the same task slices as the eval. - Then I would run a guarded online test on successful tasks per active user, keeping trust, abandonment, cost, and escalation as guardrails.

How would you A/B test an AI recommendation feature whose outputs vary even for identical inputs?

I would randomize users to stable product policies and measure the distribution of outcomes, not compare one sampled response from each model. - Assignment would persist by user or account, with model, prompt, retrieval, and sampling settings logged for every exposure. - The test would run long enough to cover repeat use, and sample size would account for higher variance from stochastic outputs. - I would predefine user-value and safety guardrails, then inspect tails and cohort harm because the same average can hide rare severe outputs.

Most requests are simple, but 15% need deeper reasoning. How would you define a small-versus-large model routing policy?

I would route by task risk and demonstrated difficulty, with a measurable fallback rule rather than by request length alone. - A small model would handle low-risk classifications that clear a calibrated confidence threshold, while regulated or tool-executing tasks go directly to the stronger model. - Ambiguous cases would escalate based on uncertainty or a cheap verifier, and the user would receive one coherent response rather than seeing model switches. - I would optimize cost per successful task and cap the large-model share, but never let the budget override the minimum quality gate for a risk class.

A company knowledge assistant can add RAG with reranking, improving answer accuracy by 9 points but adding 700 ms and $0.03 per query. Would you ship it?

I would ship reranking only for queries where the accuracy gain changes user success enough to justify its latency and cost. - I would segment gains by exact-term, ambiguous, and multi-document questions because aggregate uplift may come from a narrow class. - A staged design can retrieve broadly, rerank only uncertain queries, and bypass the stage for known navigational requests. - The decision metric would be grounded successful answers per dollar with a p95 latency guardrail, confirmed in a limited online test.

AI Product Manager interview questions

100 real questions with model answers and explanations for Middle candidates.

See a AI Product Manager resume example →

Practice with flashcards

Spaced repetition · Hunter Pass

Questions

competitivegenerics

I would start with one high-friction workflow where we can own the outcome, not with a general chat surface.

I would target the approval step that currently takes operations teams 45 minutes, because embedding into that workflow creates switching cost beyond model quality.
I would secure permission to learn from accepted edits and reviewer reasons, building proprietary task data that competitors cannot buy from a model vendor.
I would pair the workflow with distribution through the customer's existing CRM or help desk, then test whether 30% of weekly users complete the full job inside our product.

Why interviewers ask this: The interviewer is testing whether you can turn workflow, data, and distribution advantages into a defensible AI product rather than relying on model access.

I would benchmark AI against the best realistic rules-and-search flow and cap automation by the cases the product can safely own.

I would measure handle time, first-contact resolution, and correction rate for templates plus improved search before attributing any gain to AI.
I would label the share of tickets that require policy judgment or account actions; if 35% need human authority, 65% is the initial automation ceiling.
I would fund the assistant only if it beats the baseline on resolved cost per ticket while keeping factual corrections below the agreed 3% limit.

Why interviewers ask this: A strong answer separates genuine AI value from ordinary workflow improvement and recognizes that not every case should be automated.

risk-management

I would use the score to expose assumptions, but normalize value to the same time horizon before comparing the options.

I would annualize both benefits: eight saved hours per month at a $75 loaded hourly cost is $7,200 per user-year, while contract review may avoid an expected $20,000 annual loss.
I would score risk by severity and reversibility; a missed meeting detail is correctable, while a missed contract clause can create legal loss.
I would score feasibility from a 100-case eval, required integrations, and review cost, then compare risk-adjusted annual value with explicit confidence ranges.

Why interviewers ask this: The interviewer wants to see a decision tool grounded in evidence, uncertainty, and downside rather than an arbitrary weighted spreadsheet.

portfolioportfolio-hypothesishypothesis-testing

I would define a small set of portfolio beliefs that can be tested by several related bets.

One hypothesis might be that reducing analyst preparation time by 50% increases weekly completed reports, covering extraction, drafting, and review features.
I would reserve roughly 70% of capacity for the strongest workflow, 20% for an adjacent test, and 10% for cheap probes of model capabilities.
Each bet would have a shared user outcome, a model-quality gate, and a stop date so one impressive demo cannot consume the whole portfolio.

Why interviewers ask this: This tests whether you can manage correlated AI bets under uncertainty while preserving focus and learning capacity.

feedback

I would acquire data through the core workflow while preventing today's power-user behavior from defining every future user.

I would ask reviewers to confirm or correct only decisions that affect their work, capturing the original input, final label, and reason with explicit usage consent.
I would seed underrepresented industries through paid design partners and targeted annotation instead of waiting for organic volume.
I would monitor label share and error rate by customer size, language, and workflow so the flywheel does not reinforce the preferences of the loudest 10% of users.

Why interviewers ask this: A strong answer connects lawful data acquisition to product value and identifies selection bias in feedback-driven data loops.

design

I would use a tiered annotation model with clear ownership for guidelines, quality, and difficult cases.

Trained vendors can label routine intent and evidence spans, while internal support specialists adjudicate policy-sensitive or ambiguous examples.
I would pilot 300 cases, refine examples until pairwise agreement exceeds the agreed target such as Cohen's kappa of 0.75, then scale volume.
Weekly audits would sample each annotator and class, with guideline versions attached to labels so changes remain traceable.

Why interviewers ask this: The interviewer is evaluating whether you can make labeling repeatable, measurable, and appropriate to domain risk.

active-learning

I would use active learning only if selecting informative cases reduces total labeling cost without distorting the production mix.

I would compare a random-labeling curve with uncertainty or disagreement sampling, measuring quality gained per 1,000 expert dollars.
I would keep a random 20% sample because uncertainty sampling can overfocus on noisy edge cases and hide common-case regressions.
I would continue if the same release threshold is reached with materially fewer labels, for example 4,000 instead of 10,000, after including selection and review overhead.

Why interviewers ask this: A strong answer treats active learning as an economic choice with sampling risks, not as an automatic technical upgrade.

synthetic-data

I would use synthetic data to expand known scenarios, but never let it define the release truth by itself.

Synthetic cases are useful for rare formats, adversarial variations, and controlled difficulty where each expected behavior can be reviewed.
I would keep a separate human-sourced set from real workflows because generated examples inherit the generator's style, blind spots, and policy assumptions.
Release decisions would report real and synthetic slices separately, with at least a few hundred representative real cases before broad launch.

Why interviewers ask this: The interviewer wants to hear both the coverage benefit of synthetic data and its inability to represent real demand independently.

I would segment by who uses the feature, what job they are doing, and the consequence of a wrong output.

User slices would include new versus expert users, enterprise versus self-serve accounts, and every supported language with meaningful traffic.
Job slices would separate short rewrites, multi-source drafts, and policy-sensitive responses because one average hides different difficulty.
Risk slices would have independent gates, such as 95% factual pass for regulated text even if low-stakes brainstorming can ship at 85% usefulness.

Why interviewers ask this: A strong answer rejects an aggregate score that can conceal failures for important users, tasks, or risk levels.

latency

I would express each option as expected value per successful task so percentages, dollars, and seconds are not added as if they shared a unit.

Quality would have a hard floor, such as no model below 87% task success, so low cost cannot compensate for unusable output.
Above that floor, I would estimate contribution from accepted completions, then subtract inference and review cost plus the monetized completion loss caused by latency.
I would validate those estimates in an online test and publish sensitivity ranges, because routing should not hinge on one guessed dollar value for a quality point.

Why interviewers ask this: This tests whether you can combine model and product constraints without reducing the decision to one technical metric.

resilience

I would set each threshold from the cost of its false positive and false negative rather than use 0.5 for every class.

Missing fraud may cost $800 per case, so I would accept more false alerts if review capacity and customer friction remain within limits.
A false cancellation route may trigger an unwanted retention flow, so that class needs a higher precision threshold than general support.
I would calculate expected loss on a representative validation set and revisit thresholds when case value, prevalence, or reviewer cost changes.

Why interviewers ask this: A strong answer links model thresholds to asymmetric business consequences and changing operating conditions.

precisioncoverage

I would choose the highest safe coverage that keeps the uncertain queue within real human capacity.

I would plot precision against coverage by risk tier and set auto-decision thresholds separately for low-loss and high-loss cases.
At forecast volume, the abstained share must stay below 2,000 daily reviews with a buffer for peaks, vacations, and appeals.
If demand exceeds capacity, I would narrow eligible cases or improve prioritization rather than silently lower confidence and overload reviewers.

Why interviewers ask this: The interviewer is checking whether confidence policy, automation coverage, and human operations form one workable system.

launches

I would treat nDCG as a retrieval signal and require evidence that the marketplace outcome improves without damaging supply health.

The online primary metric could be completed purchases per search, with guardrails for returns, buyer cancellations, and time to first sale for new sellers.
I would segment seller exposure and concentration, because a conversion lift can come from starving the long tail rather than better matching.
A switchback or market-level test would reduce interference between treatment and control when the same inventory serves both groups.

Why interviewers ask this: A strong answer connects ranking metrics to two-sided marketplace outcomes and recognizes experiment interference.

llm-evalprogram-management

I would build a blinded, recurring program with domain-specific rubrics and an adjudication path.

Reviewers would score factual support, task completion, and harmfulness on anchored scales, with examples of what a 1, 3, and 5 mean.
Each release sample would include paired baseline and candidate answers, balanced across domains, languages, and difficult production cases.
I would track agreement, reviewer drift, and cost per evaluated case, then send disagreements to a domain owner rather than average them away.

Why interviewers ask this: The interviewer wants an operational evaluation program that produces consistent release evidence instead of occasional subjective review.

llm

I would use an LLM judge for scale only after proving where it agrees with humans and where it does not.

I would calibrate it on blinded domain-reviewer labels and report agreement by language, answer length, and risk class, not one overall correlation.
Humans would retain ownership of policy interpretation, high-impact failures, and a random audit sample because judges can favor style, position, and familiar wording.
A judge-score change would block release only with stable prompts, pinned versions, repeated runs, and a human appeal path for borderline cases.

Why interviewers ask this: A strong answer recognizes that automated judging is useful infrastructure but not an independent source of product truth.

prompting

I would give one product-engineering pair accountable ownership while distributing specific gates to domain owners.

Product owns the user-risk taxonomy and release thresholds; engineering owns deterministic execution, version capture, and blocking behavior in CI.
Applied ML owns model and prompt regressions, while legal or Trust and Safety approves the policy slices they are qualified to judge.
The RFC would name who updates cases, who may waive a failed gate, and how every waiver expires, preventing a shared system from becoming ownerless.

Why interviewers ask this: The interviewer is evaluating whether you can define clear operating ownership across technical and governance functions.

I would state the causal bridge explicitly and test each link rather than claim that seven offline points equal product value.

The hypothesis might be that better factual completion reduces edits, which cuts time to send and increases weekly completed tasks.
I would verify the first link in logged or moderated sessions, measuring edit distance and completion time on the same task slices as the eval.
Then I would run a guarded online test on successful tasks per active user, keeping trust, abandonment, cost, and escalation as guardrails.

Why interviewers ask this: A strong answer shows how offline quality becomes a testable product mechanism rather than a promotional claim.

ab-testing

I would randomize users to stable product policies and measure the distribution of outcomes, not compare one sampled response from each model.

Assignment would persist by user or account, with model, prompt, retrieval, and sampling settings logged for every exposure.
The test would run long enough to cover repeat use, and sample size would account for higher variance from stochastic outputs.
I would predefine user-value and safety guardrails, then inspect tails and cohort harm because the same average can hide rare severe outputs.

Why interviewers ask this: The interviewer wants an experiment design that treats AI output variability as part of the product rather than as measurement noise to ignore.

model-routing

I would route by task risk and demonstrated difficulty, with a measurable fallback rule rather than by request length alone.

A small model would handle low-risk classifications that clear a calibrated confidence threshold, while regulated or tool-executing tasks go directly to the stronger model.
Ambiguous cases would escalate based on uncertainty or a cheap verifier, and the user would receive one coherent response rather than seeing model switches.
I would optimize cost per successful task and cap the large-model share, but never let the budget override the minimum quality gate for a risk class.

Why interviewers ask this: A strong answer combines product policy, model economics, and risk instead of treating routing as an infrastructure-only decision.

ragrerankingqueries

I would ship reranking only for queries where the accuracy gain changes user success enough to justify its latency and cost.

I would segment gains by exact-term, ambiguous, and multi-document questions because aggregate uplift may come from a narrow class.
A staged design can retrieve broadly, rerank only uncertain queries, and bypass the stage for known navigational requests.
The decision metric would be grounded successful answers per dollar with a p95 latency guardrail, confirmed in a limited online test.

Why interviewers ask this: The interviewer is testing product judgment about the quality, latency, and cost trade-off of a concrete RAG architecture choice.

Locked questions

21
A prompt-and-RAG system still writes in the wrong domain style. How would you decide whether to fine-tune, including its maintenance cost?
ragpromptingfine-tuning
22
Policies change daily, but rebuilding the knowledge index takes six hours. How would you define freshness and SLA requirements for the product?
indexes
23
An AI assistant is useful but sometimes invents details. What hallucination and trust strategy would you put on the roadmap?
hallucinationroadmaproadmapping
24
How would you design AI UX for uncertainty without showing users a misleading confidence percentage?
design
25
Users often fix AI drafts but rarely click thumbs down. How would you collect correction and feedback data without adding heavy friction?
feedback
26
An AI workflow sometimes needs a specialist. How would you design human escalation as part of the product rather than an exception?
escalationdesignerror-handling
27
A vendor releases a new model version every few months. What lifecycle and version-acceptance process would you own?
procurementreleasesconcurrency
28
How would you use shadow and canary stages to launch a new model policy without waiting for perfect certainty?
launchesdeployment-strategies
29
What product-level drift monitoring would you require for an AI extraction feature after launch?
monitoringiaclaunches
30
At 09:00, acceptance of an AI support agent falls from 72% to 51% after the model, prompt, retrieval corpus, refund policy, and review screen all changed in one release. How would you design observability so Product can attribute the regression before the 15:00 incident review?
retrievalpromptingagents
31
Usage may grow from 100,000 to two million AI tasks per month. How would you build an inference budget and forecast?
inference
32
The team proposes semantic caching and speculative decoding to lower AI cost and latency. What product trade-offs would you evaluate?
cachinglatencysemantic-cache
33
How would you choose a pricing metric and packaging for an AI research workflow with highly variable usage?
pricingmonitoring
34
An AI workflow brings $40 per account monthly but inference, review, and support costs vary widely. How would you manage gross margin by workflow?
cssinference
35
A few customers consume 20 times the median AI usage on an unlimited plan. How would you design usage caps or a fair-use policy?
design
36
An enterprise buyer asks for a 99.9% SLA on an AI drafting feature. What would you include and exclude from the commitment?
37
What customer-data isolation requirements would you set for a multi-tenant enterprise RAG product?
ragmulti-tenancy
38
How would you define data retention and privacy for prompts, outputs, feedback, and eval traces?
promptingfeedbackretention
39
When and how would you run a Trust and Safety review for a new AI coaching feature?
40
A red-team exercise finds jailbreaks, biased refusals, and unsafe tool choices. How would you turn those findings into roadmap input?
llm-safetyred-teamroadmap
41
How would you design risk-tiered controls for one AI platform serving brainstorming, customer support, and payment actions?
risk-managementdesign
42
What would you include in a customer-facing model card or AI feature disclosure?
model-card
43
What kill criteria would you set before launching an AI feature pilot?
launches
44
How would you sequence a roadmap when a needed model capability may improve in six months but is unreliable today?
roadmaproadmapping
45
How would you write OKRs that link model performance to user and business outcomes for an AI support assistant?
performance
46
Applied research wants three months to explore a new agent architecture, while product needs a committed customer workflow. Where would you set the partnership boundary?
agentsarchitecture
47
What should a cross-functional RFC contain before building an enterprise AI decision-support feature?
cross-functionalcross-teamdecision-making
48
How would you enable sales to sell an AI feature without overclaiming quality or future capability?
49
How would you run customer discovery for an enterprise AI workflow when buyers, administrators, and daily users want different things?
50
An AI writing feature will launch in six languages and must work with assistive technology. How would you set accessibility and localization quality requirements?
launchesa11y
51
A claims classifier reaches 82% precision, but only 31% of adjusters say they trust its recommendations after a two-week pilot of 400 cases. What do you decide before the rollout meeting on Friday?
precisionreleases
52
A medical intake model has 91% overall accuracy but misses 12% of urgent cases, while the current manual process misses 4%; launch is scheduled in 10 days. What do you do?
launchesconcurrency
53
One fraud threshold catches 88% of abuse but blocks 6% of legitimate new customers and only 0.8% of established customers; Finance needs a decision in five days. Would you keep one threshold?
54
A new writing model lifts rubric accuracy from 84% to 90%, but user task completion falls from 68% to 59% during a seven-day test with 8,000 sessions. Which model ships next Monday?
sessions
55
A sales copilot reaches 64% weekly adoption, but audited hallucinations rose from 1.5% to 4.8% across 600 outputs and renewal calls begin in three weeks. Do you keep promoting it?
hallucinationdecision-making
56
Adding citations to a research assistant raises trusted-answer ratings from 52% to 73%, but p95 latency rises from 3.2 to 7.1 seconds and completion drops 6 percentage points; a launch decision is due Thursday. What do you choose?
latencycitationslaunches
57
After tightening an evidence gate, unsupported answers fall from 5% to 1%, but abstention rises from 9% to 38% and support deflection falls 18% in one week. What do you change by Friday?
abstention
58
An AI review queue receives 1,200 flagged cases per day, but the operations team can review only 700 and the backlog will exceed 5,000 by Monday. What product decision do you make in the next 48 hours?
backlogdata-structuresagile
59
A model candidate gains 4 points on new customer examples but regresses 7 points on the 2,000-item golden set two days before release. Engineering says the old set is stale. Do you waive the regression?
releases
60
A synthetic eval predicts a 9-point quality gain, but a 300-case human review finds only a 1-point gain and twice as many tone violations; launch is in six days. Which evidence wins?
launches
61
An LLM judge scores a summarizer at 92%, while three trained reviewers score it at 74% and disagree with the judge on 23% of 400 cases. You need a go or no-go decision tomorrow. What do you do?
llmconflict
62
Labeling 5,000 domain examples will cost $120,000 and take six weeks, while a cheaper vendor offers $35,000 in three weeks with 71% inter-annotator agreement. What do you recommend by Tuesday?
annotator-agreementprocurement
63
A lending assistant's approval recommendation accuracy falls from 87% to 79% for applications received after a policy change 12 days ago, while older traffic remains stable. What do you decide within 24 hours?
64
Your ranking model learns from user clicks, but after four weeks it shows the same popular templates 62% of the time and new-template discovery falls from 28% to 9%. What change do you approve this sprint?
agile
65
A RAG reranker lifts grounded-answer accuracy from 76% to 84%, but adds 850 ms at the median and raises p95 latency from 2.9 to 4.6 seconds; conversion historically drops above four seconds. What ships in eight days?
ragrerankinggroundedness
66
A policy assistant cited a document deleted 36 hours earlier in 27 customer sessions, and the index freshness SLA is four hours. What do you decide before noon today?
indexessessions
67
A fine-tuned support model improves resolution accuracy from 78% to 86%, but requires $90,000 upfront and about $18,000 per month for new labels and retraining; the base model is updated quarterly. What do you recommend in five days?
fine-tuning
68
After a foundation-model update at 09:00, completed bookings fall 24%, tool-call errors rise from 2% to 15%, and 18,000 sessions are exposed. What is your product decision in the next 30 minutes?
sessions
69
A vendor will repoint the alias your product uses to a new model in 72 hours; your 600-case eval shows a 5-point quality loss on refunds but a 3-point gain overall. What do you decide today?
procurement
70
Routing 70% of low-stakes requests to a small model saves $160,000 per month, but accepted-output rate falls from 74% to 68% in a 20,000-session test. Do you launch next week?
launchessessions
71
Month-end inference spend is forecast at $520,000 against a $400,000 budget, with 12 days remaining and no evidence of fraud or outage. What do you cut by tomorrow?
inference
72
A richer model raises answer acceptance by 5 percentage points, but p95 latency increases from 3.5 to 6.8 seconds and checkout conversion falls from 11.2% to 9.6% over 14 days. What do you ship Friday?
latency
73
Semantic caching could save $85,000 per month, but 6% of cached policy answers may remain stale for up to 24 hours; Procurement needs a decision this week. What do you approve?
cachingsemantic-cacheprocurement
74
A vendor's speculative decoding feature cuts p95 latency by 35% and could lift conversion 4%, but requires a two-year exclusive serving contract worth $2.4M. What recommendation do you make in 10 days?
latencyprocurementdecoding
75
An AI support feature costs $0.42 per conversation and appears to deflect 40% of contacts, but only 58% of that group report their issue resolved after 30 days. Which unit metric do you use for the next quarter?
monitoring
76
Your $19 monthly AI plan generates $24 in average inference and support cost for heavy users, who represent 18% of subscribers; pricing review is in seven days. What do you recommend?
pricinginference
77
A new 500-generation monthly cap would reduce inference spend by $110,000, but 900 power users generated 28% of referrals and 41% say they will leave; rollout is planned in 14 days. What do you change?
releasesinference
78
An enterprise customer requests a 99.9% monthly availability SLA for an AI workflow, but current availability is 99.4% and the contract worth $1.8M must be signed in three weeks. What do you promise?
promises
79
One tenant with $900,000 in annual recurring revenue sees only 71% extraction accuracy while the other 24 enterprise tenants average 89%, and renewal is in 45 days. Do you prioritize a tenant-specific fix?
prioritization
80
An AI account analyst is due to launch in eight days. English scores 91% with $2.4M contracted revenue and daily reviewer coverage; French scores 87% with $900,000 and coverage three days a week; Japanese scores 82% with a $1.6M customer commitment and no qualified reviewer for six weeks; Portuguese scores 86% with $150,000 in pipeline and coverage two days a week. The launch gate is 85%. Which locales do you launch?
launcheslaunch-gatecoverage
81
Five days before launch, accessibility testing shows screen-reader users cannot review or correct 38% of AI-generated form fields, affecting about 6,000 monthly users. Do you launch the feature?
a11yformstesting
82
An AI project coordinator was randomized by user across 24,000 members in 600 shared workspaces. After 21 days, treated users complete 13% more tasks, but their generated assignments and summaries are visible to control teammates, and 68% of workspaces contain both groups. The board readout is in 48 hours. How do you interpret and redesign the test?
design
83
After 12 days, an AI onboarding experiment shows a 3-percentage-point activation lift, but it has only 1,800 of the 8,000 users required for 80% power; the launch slot closes Friday. What do you decide?
experimentsonboardingactivation
84
Daily use of a new AI coach jumps to 44% in week one but falls to 19% by week four, below the old feature's 23%; the team wants a full launch next week based on sign-ups. What do you do?
launches
85
A trust survey improves from 3.1 to 4.0 out of 5 after adding confidence labels, but users verify only 14% of high-risk answers versus 26% before; a decision is due in four days. Which signal matters?
risk-management
86
Mandatory human review cuts severe AI errors from 3.2% to 0.7%, but 46% of users abandon during a median six-hour wait; renewal analysis is due next Wednesday. What do you change?
87
Legal finds that 35% of 200 pilot users did not understand their support chats would train the model, and public launch to 50,000 users is in nine days. What do you ship?
launches
88
A red-team run finds that 8% of 500 adversarial prompts produce instructions for bypassing workplace safety controls; beta expansion from 2,000 to 20,000 users is scheduled in 48 hours. What do you decide?
promptingred-team
89
A support assistant invented a refund policy in 63 conversations over 11 hours, leading to $18,000 in disputed promises. What do you do in the next two hours and over the next seven days?
promises
90
An AI meeting-summary feature has 12% weekly adoption after three months, costs $70,000 per month, and saves active users only four minutes per week; Q3 planning starts Friday. Do you keep it?
decision-making
91
Your 10-week quarter has 18 engineer-weeks available: the promised eval dashboard needs 10, an inference-cost project needs 8 and saves $300,000 quarterly, and a sales-requested agent demo needs 7 for a conference in six weeks. What do you prioritize Monday?
promisesagentsinference
92
Applied Research presents an impressive agent prototype after 8 weeks, but it has no eval set, no baseline, and needs 6 engineer-weeks to productize before a customer demo in 1 month. What do you decide this week?
agentsprototypes
93
Sales promised 95% extraction accuracy to a $1.2M prospect closing in 20 days, but the product averages 86% and the prospect's 150-document sample scores 81%. How do you respond by tomorrow?
promises
94
A $1.1M enterprise customer supplies an 800-case domain eval set and asks for 93% overall accuracy to become a contractual acceptance term in 10 days. Yet 46% of cases are easy renewals that represent only 14% of production, and two customer reviewers disagree on 19% of labels. What do you negotiate?
conflict
95
The launch scorecard requires hallucinations below 2%, but the latest 1,000-case eval is 2.8%; the CMO asks for a waiver because a campaign starts in 36 hours. What do you decide?
hallucinationlaunches
96
A 10% canary of a model release shows task completion down 8 percentage points, p95 latency up 22%, and 4,500 exposed sessions after three hours. Engineering wants another day of data. Do you roll back now?
latencysessionsrollback
97
Your quarterly AI scorecard shows quality up 6 points and inference cost down 18%, but successful task completion fell 5% and support contacts rose 12%; the business review is Friday. How do you judge performance?
inferenceperformance
98
At 15:00, the CEO asks for a one-page update by 17:00 after AI adoption missed the 40% target at 24%, inference spend exceeded plan by $210,000, and hallucinations stayed within the 2% gate. What do you send?
hallucinationdecision-makinginference
99
An APM proposes launching an AI reply feature to 100% of users in three weeks based on 20 demo prompts and no failure taxonomy; you have a 45-minute coaching session tomorrow. How do you mentor them?
promptingmentoringsessions
100
Your RFC recommends routing only low-risk requests to a cheaper model, while the ML lead wants 80% routing to hit a $250,000 quarterly savings target; the architecture review is in 48 hours. How do you resolve the disagreement?
conflictdecision-makingarchitecture

Questions

1A generic AI copilot can be copied by competitors within one quarter. How would you choose a product wedge that can become a durable moat?

2A support team resolves tickets in 14 minutes using templates and search. How would you set the non-AI baseline and the automation ceiling before funding an AI assistant?

3You must choose between an AI meeting summarizer and an AI contract reviewer for the next quarter. How would you apply a value, risk, and feasibility score?

4Leadership has five AI ideas but only two squads for six months. How would you frame a portfolio hypothesis instead of ranking isolated features?

5A document classifier needs proprietary labels, but only current power users generate feedback. What data acquisition and flywheel strategy would you choose?

6Your team needs 20,000 labeled support conversations across six issue types. How would you design the annotation operating model?

7Experts cost $12 per labeled case, and the model sees one million unlabeled cases monthly. When does active learning make product-economic sense?

8The team proposes generating 80% of a new eval set synthetically because real examples are scarce. What role would you allow synthetic data to play?

9An AI drafting feature scores 88% overall. How would you segment the eval before deciding whether that number is acceptable?

10Model A has 92% quality, $0.18 cost, and 4.5-second latency; model B has 88% quality, $0.04 cost, and 1.4-second latency. How would you define a product utility function?

11A classifier routes billing, cancellation, fraud, and general support tickets. How would you set per-class thresholds from business loss?

12An AI reviewer can auto-decide 70% of cases at 96% precision, while humans can review only 2,000 cases per day. How would you choose confidence and coverage targets?

13A marketplace ranking model improves nDCG by 6% offline but may concentrate demand among top sellers. What would you require before launch?

14How would you set up a human evaluation program for an enterprise answer assistant used across finance, HR, and operations?

15The team wants an LLM judge to replace most human review. What limits would you set from a product-management perspective?

16Who should own eval-as-CI when model, prompt, data, and product teams all contribute to an AI feature?

17Offline task success rises from 82% to 89%, but what bridge would you require before predicting an online user outcome?

18How would you A/B test an AI recommendation feature whose outputs vary even for identical inputs?

19Most requests are simple, but 15% need deeper reasoning. How would you define a small-versus-large model routing policy?

20A company knowledge assistant can add RAG with reranking, improving answer accuracy by 9 points but adding 700 ms and $0.03 per query. Would you ship it?

Locked questions