A subscription service has 500,000 active customers, 6% monthly churn, a $90 retained margin, and a $12 retention offer. An experiment says the offer prevents 25% of churn, but operations can contact only 40,000 customers per month. Which metric and decision threshold do you use?

I would optimize incremental net profit at the 40,000-customer capacity, not accuracy or a generic F1 score. - With a constant 25% treatment effect, contact when p(churn) × 0.25 × $90 exceeds $12, giving a break-even churn probability of 53.3%. - Because the 40,000-contact limit may bind first, I rank customers by expected incremental profit and take only positive-value customers up to the cap. - I report profit@40k, uplift@40k, and AUUC from a randomized holdout; ROC AUC is secondary because it ignores treatment effect and campaign economics. - If treatment effects vary, I replace the churn ranking with a CATE model such as a causal forest and keep an untreated holdout to verify realized incremental margin.

A payment processor scores 10 million transactions per day; confirmed fraud is 0.15%, a missed fraud costs $900, a manual review costs $6, and reviewers can inspect 5,000 transactions daily. How would you choose the metric and review threshold?

I would rank by expected loss avoided and optimize recall at the fixed 5,000-review budget. - For a calibrated fraud probability p, review has positive gross value when p × $900 exceeds $6, so the unconstrained threshold is 0.67%. - Capacity is tighter, so the operational threshold is the score of the 5,000th transaction among cases above 0.67%; I would recompute it daily as the score distribution moves. - I use recall@5k, precision@5k, and dollars saved per 1,000 reviews; PR AUC is diagnostic, while ROC AUC can look excellent at a 0.15% base rate. - I validate on a time-based sample with mature chargeback labels and bootstrap confidence intervals for net savings and recall.

A hospital validates a screening model on 20,000 patients with 2% disease prevalence. A false negative has an estimated $25,000 clinical cost, a false positive triggers a $400 test, and the clinic can handle at most 300 follow-ups per 5,000 weekly screenings. What operating point would you recommend?

I would choose the highest-sensitivity threshold that stays within 300 weekly referrals and passes a prespecified safety floor. - Pure expected cost gives p > $400 / ($25,000 + $400) = 1.57%, but I would not use that number alone for a clinical decision. - On the 20,000-patient validation set, I would require at least 95% sensitivity with a one-sided 95% bootstrap lower bound, then check whether predicted positives stay below 6% of screenings. - I report sensitivity, NPV, referrals per 1,000, and subgroup sensitivity by age, sex, and site; accuracy is misleading at 2% prevalence. - If no threshold satisfies both the safety floor and the 300-case capacity, I escalate the capacity conflict rather than silently trade away missed cases.

An e-commerce search team has an offline set of 2 million queries, a 1.2% purchase rate, an average $65 order margin, and only the top 10 results are visible without scrolling. The reranker must stay below 80 ms p95 latency. Which ranking metric and launch bar do you set?

I would make propensity-corrected NDCG@10 the primary offline metric and require a business lift without breaking latency. - NDCG@10 rewards putting graded outcomes such as purchase, add-to-cart, and click near the top; MRR would overvalue only the first relevant result. - I require at least a 1.5% NDCG@10 lift with a 95% paired-bootstrap interval above zero, no more than a 0.2% drop in query coverage, and p95 latency below 80 ms. - Sparse purchases get the highest gain, but I debias click labels with inverse propensity weighting and inspect NDCG separately for head and tail queries. - The online gate is positive margin per query with its 95% interval above zero; NDCG is a proxy, so it cannot authorize launch by itself.

A recommender serves 1 million weekly users, 3% of whom purchase, with 20 recommendation slots and about $4 of lost margin for an irrelevant occupied slot. Product requires the top 1% of items to receive no more than 35% of impressions, and scoring has a 50 ms p95 limit. How do you evaluate a candidate model?

I would use NDCG@20 for relevance, then enforce concentration and latency as hard guardrails. - NDCG@20 captures both ranking position and graded value, while Recall@20 checks whether the model retrieves any of a user's held-out relevant items. - On a temporal user-level holdout, I require at least +2% NDCG@20 and no Recall@20 regression, with paired-bootstrap intervals rather than treating interactions as independent rows. - I reject the model if the top 1% exceeds 35% of impressions, and I also report catalog coverage, intra-list diversity, and novelty to expose popularity collapse. - The online decision requires at least +0.3% margin per user with a 95% interval above zero and p95 scoring below 50 ms, because offline relevance does not price every bad slot correctly.

A lender has a 100,000-loan validation sample with a 4% default rate. Approving a future defaulter loses $8,000, rejecting a good borrower loses $900 of margin, and policy requires approving at least 50% of applicants. A model has ROC AUC 0.86 but visibly overpredicts risk. How do you calibrate and set the cutoff?

I would calibrate probabilities first, then minimize expected cost subject to the 50% approval floor. - For a calibrated default probability p, reject when p × $8,000 exceeds (1 - p) × $900, giving a cost-only cutoff of 10.1%. - I fit Platt scaling and isotonic regression on separate temporal calibration folds, choose by out-of-fold Brier score, and keep isotonic only if its extra flexibility is stable. - I inspect reliability curves and Brier reliability, resolution, and uncertainty components overall and by score band; ROC AUC cannot reveal probability bias. - If the 10.1% cutoff approves less than 50%, I quantify the extra expected loss at the policy-constrained cutoff instead of presenting it as model-optimal.

An ad model is evaluated on 5 million recent impressions with a 0.8% click rate. A click is worth $1.20, auction prices range from $0.002 to $0.03 per impression, and daily spend cannot exceed $100,000. Offline AUC improved, but reliability plots show underprediction above 2% CTR. What do you do?

I would recalibrate pCTR and bid from expected value, because ranking quality alone does not make bids economically correct. - I fit Platt scaling or beta calibration on a recent time split, selecting by Brier score and log loss, then inspect equal-frequency reliability bins by device and placement. - For an impression priced at c, the break-even rule is calibrated pCTR > c / $1.20; a $0.012 impression therefore needs pCTR above 1.0%. - I rank eligible impressions by expected surplus pCTR × $1.20 - c and raise the minimum surplus threshold or scale bids down until projected spend fits the $100,000 budget. - I require lower Brier score, reliability error below 0.2 percentage points in material bins, and no loss in value-weighted PR AUC before an online revenue test.

A content platform reviews 300,000 uploads per day; 0.4% contain severe policy violations, a missed violation is valued at $200 of harm, a false escalation costs $8, and moderators can review 12,000 items daily. Which metric and threshold would you use?

I would minimize expected harm under the 12,000-review cap, with recall at capacity as the main operational metric. - With calibrated probabilities, review when p × $200 exceeds (1 - p) × $8, which gives a cost-only threshold of 3.85%. - If more than 12,000 items clear 3.85%, I take the top 12,000 by expected harm; if fewer do, I do not fill the queue with negative-value cases. - I report recall@12k, precision@12k, PR AUC, and expected harm per 10,000 uploads; 99.6% accuracy from predicting safe is useless. - I bootstrap by uploader, not item, and require recall floors for high-severity categories so a strong aggregate score cannot hide a dangerous subgroup.

A retailer forecasts 14 million SKU-store-day rows with 35% zero demand. Underforecasting one unit loses $6 of margin, overforecasting one unit costs $1.50 in markdowns, replenishment runs daily, and the final backtest covers 8 weeks. Which forecast and evaluation metric would you use?

I would optimize an 80th-quantile demand forecast with business-weighted pinball loss rather than minimize RMSE. - The cost-optimal quantile is $6 / ($6 + $1.50) = 0.80, so the prediction deliberately carries more stock than a median forecast. - I report realized dollar loss, q80 pinball loss, signed unit bias, and service level by SKU-volume band; one aggregate WAPE can hide failures on intermittent items. - The 8-week backtest uses rolling daily origins and only inventory, promotion, and calendar data known before each replenishment cutoff. - I compare gradient boosting with seasonal-naive and Croston-style baselines, and keep the simpler method wherever its dollar loss is statistically indistinguishable.

On an 80,000-customer holdout with 7% churn, model A has ROC AUC 0.91 and Brier score 0.064, while model B has ROC AUC 0.88 and Brier score 0.049. A retention contact costs $10, preserves $120 margin in 20% of true churners, and is capped at 8,000 contacts. Observed holdout profit@8k is $39,000 for A and $52,000 for B. Which model do you ship?

I would choose model B for this campaign, provided the profit difference is statistically and temporally robust. - The break-even churn probability is $10 / (0.20 × $120) = 41.7%, then the 8,000-contact cap selects the highest expected-profit eligible customers. - B wins the decision metric, profit@8k, by $13,000 and has better probability accuracy despite lower ROC AUC; A's stronger global ordering is not the business objective. - I use paired customer-level bootstrap intervals for the $13,000 difference and inspect reliability curves around 30% to 60%, where contact decisions change. - I would also compare isotonic recalibration of A on a separate fold, but ship A only if recalibration raises validated profit@8k above B without unstable bins.

You receive 12.4 million loan applications from 18 lenders dated 2021-01-01 through 2025-06-30; 3.1% become 90-day defaults. The extract includes collections_status and current_days_past_due, both refreshed after approval, and scoring must happen at approval for an October 2025 launch. What exact train, validation, and test split would you use, which data would you exclude, and why?

I would use approval-time chronological cohorts and reconstruct every feature as it existed when the lending decision was made. - I would train on approvals from 2021-01-01 through 2023-12-31, validate on 2024-01-01 through 2024-06-30, and lock 2024-07-01 through 2025-06-30 as the test set after all 90-day outcomes mature. - I would remove collections_status, current_days_past_due, repayment totals, charge-off fields, and any bureau refresh received after approval rather than trusting their column names. - I would require each feature timestamp to be at or before approval time and reproduce the join from immutable source history; a feature without historical availability evidence would not enter the model. - I would tune only on validation, open the test once, and report pooled and per-lender results so a lender-mix change cannot hide leakage or weak generalization.

A churn table contains 73 million user-day rows from 9.6 million users in six markets between 2022-01-01 and 2025-12-31; the next-30-day churn rate is 2.4%, seasonality is strong, and the model will be retrained monthly to score January and February 2026. As of 2026-01-15, what exact split would you use and why?

I would validate forward in time with 30-day purge windows, because a random row split would mix seasons and overlapping outcome windows. - I would train on feature dates from 2022-01-01 through 2024-05-31, purge June 2024, and validate on 2024-07-01 through 2024-11-30. - I would purge December 2024 and lock 2025-01-01 through 2025-11-30 as the test period; December 2025 rows stay out because their 30-day labels are not all mature on 2026-01-31. - I would fit preprocessing only on the training period and score every row using the feature snapshot available on that row's date, without carrying future category frequencies or imputations backward. - I would report each month and each of the six markets separately, then weight them by the expected January and February 2026 traffic mix rather than trusting one aggregate score.

You have 2.1 million chest radiographs from 94,000 patients across 12 hospitals, collected from 2022-01-01 through 2025-06-30; 8.7% of patients are pneumonia-positive, many patients have repeated studies, and compute allows only five full training runs. What exact cross-validation split would you use and why?

I would run five-fold stratified grouped cross-validation with patient_id as the indivisible group. - I would assign every image from one patient to exactly one fold, so prior and follow-up studies can never appear on opposite sides of a split. - I would construct five roughly 20% patient folds with seed 20250630, balancing the patient-level label and hospital counts as closely as the group constraint permits. - Each run would train on four folds and predict the fifth, producing one out-of-fold prediction per patient within the five-run budget. - I would aggregate metrics at patient level and report them by hospital as well as overall, because image-level confidence intervals would overstate the effective sample size.

A credit dataset has 240,000 unique applicants acquired through eight channels from 2025-01-01 through 2025-06-30; the default base rate is 4.8%, policy and population were stable during the period, one stratified split gives visibly unstable PR-AUC, and the budget is 25 full fits. What exact repeated-validation split would you use and why?

I would reserve one untouched test set and use repeated stratified cross-validation on the remaining applicants to measure split sensitivity. - I would place a deterministic stratified 20% of applicants in the locked test set, balancing default status and the eight acquisition channels, and keep the other 80% for development. - On development data I would run five-fold stratified cross-validation four times with seeds 11, 29, 47, and 83, for 20 fits with different fold assignments. - I would compare fixed candidate settings using mean PR-AUC, its dispersion across the 20 folds, and channel-level recall rather than selecting the highest single fold. - After choosing the setting, I would refit once on all development data and evaluate once on the locked test, using 22 of the 25 available full fits.

You must compare four model families and 60 total hyperparameter configurations on 78,000 pathology slides from 15,600 patients at nine labs, collected from 2020-01-01 through 2024-12-31; 6.4% of patients are positive, only 410 positives belong to the critical subtype, no separate holdout can be spared, and the budget is 1,220 fits. What exact nested cross-validation scheme would you use and why?

I would use patient-grouped nested cross-validation so model selection and performance estimation never use the same held-out patients. - The outer loop would be five-fold StratifiedGroupKFold by patient_id, with label and lab distributions balanced across the five roughly 20% patient folds. - Inside each outer training partition, a four-fold StratifiedGroupKFold would rank all 60 configurations by patient-level PR-AUC while keeping every patient's slides together. - I would refit the inner winner on the full outer training partition and score the untouched outer fold, consuming 1,200 inner fits plus five outer refits. - I would report the distribution of the five outer scores, including critical-subtype sensitivity, then select a configuration by its inner-loop ranks and use one final fit on all 15,600 patients.

You have six million account-month examples derived from 900 million events at 14 lenders between 2022-01-01 and 2025-06-30; 1.9% enter 60-day delinquency, warehouse tables expose current corrected values, and month-end scoring must finish within four hours. What exact split and point-in-time feature construction would you use and why?

I would split by scoring cutoff and rebuild each feature from records that were genuinely available by that cutoff. - I would train on cutoffs from 2022-01-31 through 2023-12-31, validate on 2024-01-31 through 2024-06-30, and test on 2024-07-31 through 2025-04-30; May and June 2025 stay out because their 60-day labels are immature. - Every join would require event_time at or before the cutoff and valid_from at or before the cutoff with valid_to after it, using append-only history rather than the current corrected row. - I would discard a feature if its historical versions cannot be reconstructed, and I would build rolling aggregates with windows ending at the cutoff rather than at extraction time. - I would fit encoders and imputers on the training cutoffs only, then replay the complete four-hour month-end computation for validation and test and report results by lender.

An advertising dataset contains 1.8 billion impressions from 12 countries between 2024-01-01 and 2025-12-31; 0.42% lead to a purchase within 30 days, labels arrive continuously, and weekly retraining may wait only seven days after an impression. As of 2026-01-31, what exact split would you use and how would you prevent delayed labels from corrupting validation?

I would evaluate only on eventually mature cohorts while reproducing the seven-day label view available at every weekly training cutoff. - I would train on impressions from 2024-01-01 through 2025-06-30, validate on 2025-07-01 through 2025-09-30, and lock 2025-10-01 through 2025-12-31 as test, whose 30-day outcomes are mature by 2026-01-31. - For each simulated weekly fit, impressions aged 7 to 29 days would remain censored rather than being marked as non-purchases; rows at least 30 days old supply complete labels. - If recent rows are needed, I would estimate conversion-delay probabilities from mature training cohorts and apply a fixed censoring correction learned without validation or test outcomes. - I would score validation and test against their final 30-day labels by country and impression week, so different reporting delays or country mixes cannot create an artificial gain.

A marketplace has 220 million interactions from 4.6 million users with 680,000 items sold by 38,000 sellers in 22 categories between 2023-01-01 and 2025-12-31; the purchase rate is 0.7%, the launch target is brand-new sellers, and the final holdout must contain at least 10,000 purchases. What exact cold-start split would you use and why?

I would hold out sellers by onboarding date and evaluate only their first 14 days, because random interactions would let seller identity and prior engagement leak into cold-start evaluation. - Training would use interactions through 2025-06-30 from sellers onboarded before 2025-07-01, with all vocabularies, popularity priors, and preprocessing fitted on that population. - Validation would contain the first 14 days for sellers first onboarded from 2025-07-01 through 2025-09-30, and it alone would drive feature and threshold choices. - Test would contain the first 14 days for sellers first onboarded from 2025-10-01 through 2025-12-17, leaving enough calendar time for outcomes through 2025-12-31 and verifying the 10,000-purchase minimum before locking the split. - Seller IDs, item interaction histories, and post-onboarding aggregates for validation and test sellers would be unavailable, while listing content and category fields present at onboarding would remain allowed.

A wildfire dataset has 3.2 million cell-month rows from 2010-01-01 through 2024-12-31 across 18 regions labeled R01 through R18; fires occur in 1.3% of rows, residual correlation extends about 25 km, deployment is specifically to unseen regions R16 through R18, and only four validation fits are affordable. What exact spatial split would you use and why?

I would make R16 through R18 a locked spatial test and tune with four contiguous regional folds inside R01 through R15. - No row from R16, R17, or R18 would enter training, preprocessing, or threshold selection, and I would remove training cells within 25 km of their borders to limit spatial bleed. - The four validation folds would be contiguous blocks R01-R04, R05-R08, R09-R12, and R13-R15, with each run training on the other three blocks across the full 2010-2024 period. - Weather normalization, missing-value rules, and spatial aggregates would be fitted within each training side, and neighbor features could not cross the held-out block boundary. - I would select the fixed setting from the four validation results, refit on eligible R01-R15 cells, and report R16, R17, and R18 separately as well as pooled.

A demand-forecasting table has 96 million SKU-store-day rows for 38,000 SKUs and 2,400 stores from 2019-01-01 through 2025-12-31; 17% of rows have zero demand, the horizon is 28 days, promotions are known only 14 days ahead, and compute permits six full backtest fits after settings are narrowed on smaller samples. What exact temporal backtest would you run and why?

I would run six expanding-window forecast origins and preserve exactly the information available at each origin. - The origins would be 2024-07-01, 2024-10-01, 2025-01-01, 2025-04-01, 2025-07-01, and 2025-10-01, each scored on the following 28 calendar days. - For each origin, training would start on 2019-01-01 and include only examples whose full 28-day target window ends before that origin, preventing target overlap with the forecast window. - Promotion values would be used as known for days 1 through 14 and masked or replaced by the planning-time fallback for days 15 through 28, never filled from realized promotions. - I would use the first five origins for the final setting decision and open the October 2025 origin once as the locked result, reporting weighted error by store, SKU, and zero-demand cohort.

Data Scientist interview questions

100 real questions with model answers and explanations for Senior candidates.

See a Data Scientist resume example →

Practice with flashcards

Spaced repetition · Hunter Pass

Questions

retentionchurnmonitoring

I would optimize incremental net profit at the 40,000-customer capacity, not accuracy or a generic F1 score.

With a constant 25% treatment effect, contact when p(churn) × 0.25 × $90 exceeds $12, giving a break-even churn probability of 53.3%.
Because the 40,000-contact limit may bind first, I rank customers by expected incremental profit and take only positive-value customers up to the cap.
I report profit@40k, uplift@40k, and AUUC from a randomized holdout; ROC AUC is secondary because it ignores treatment effect and campaign economics.
If treatment effects vary, I replace the churn ranking with a CATE model such as a causal forest and keep an untreated holdout to verify realized incremental margin.

Why interviewers ask this: This tests whether the candidate can turn churn probabilities, treatment effect, unit economics, and a capacity limit into a decision rule.

transactionsmonitoringconcurrency

I would rank by expected loss avoided and optimize recall at the fixed 5,000-review budget.

For a calibrated fraud probability p, review has positive gross value when p × $900 exceeds $6, so the unconstrained threshold is 0.67%.
Capacity is tighter, so the operational threshold is the score of the 5,000th transaction among cases above 0.67%; I would recompute it daily as the score distribution moves.
I use recall@5k, precision@5k, and dollars saved per 1,000 reviews; PR AUC is diagnostic, while ROC AUC can look excellent at a 0.15% base rate.
I validate on a time-based sample with mature chargeback labels and bootstrap confidence intervals for net savings and recall.

Why interviewers ask this: This checks rare-event evaluation, cost-sensitive thresholding, and the effect of a hard review-capacity constraint.

estimationvalidation

I would choose the highest-sensitivity threshold that stays within 300 weekly referrals and passes a prespecified safety floor.

Pure expected cost gives p > $400 / ($25,000 + $400) = 1.57%, but I would not use that number alone for a clinical decision.
On the 20,000-patient validation set, I would require at least 95% sensitivity with a one-sided 95% bootstrap lower bound, then check whether predicted positives stay below 6% of screenings.
I report sensitivity, NPV, referrals per 1,000, and subgroup sensitivity by age, sex, and site; accuracy is misleading at 2% prevalence.
If no threshold satisfies both the safety floor and the 300-case capacity, I escalate the capacity conflict rather than silently trade away missed cases.

Why interviewers ask this: This tests whether business costs inform a medical threshold without overriding safety, uncertainty, or subgroup checks.

csslatencyqueries

I would make propensity-corrected NDCG@10 the primary offline metric and require a business lift without breaking latency.

NDCG@10 rewards putting graded outcomes such as purchase, add-to-cart, and click near the top; MRR would overvalue only the first relevant result.
I require at least a 1.5% NDCG@10 lift with a 95% paired-bootstrap interval above zero, no more than a 0.2% drop in query coverage, and p95 latency below 80 ms.
Sparse purchases get the highest gain, but I debias click labels with inverse propensity weighting and inspect NDCG separately for head and tail queries.
The online gate is positive margin per query with its 95% interval above zero; NDCG is a proxy, so it cannot authorize launch by itself.

Why interviewers ask this: This assesses metric choice for ranked lists, sparse outcomes, position bias, and explicit launch constraints.

decision-makingcss

I would use NDCG@20 for relevance, then enforce concentration and latency as hard guardrails.

NDCG@20 captures both ranking position and graded value, while Recall@20 checks whether the model retrieves any of a user's held-out relevant items.
On a temporal user-level holdout, I require at least +2% NDCG@20 and no Recall@20 regression, with paired-bootstrap intervals rather than treating interactions as independent rows.
I reject the model if the top 1% exceeds 35% of impressions, and I also report catalog coverage, intra-list diversity, and novelty to expose popularity collapse.
The online decision requires at least +0.3% margin per user with a 95% interval above zero and p95 scoring below 50 ms, because offline relevance does not price every bad slot correctly.

Why interviewers ask this: This checks whether the candidate balances ranking quality, user-level uncertainty, diversity, business value, and serving constraints.

evaluationvalidationcss

I would calibrate probabilities first, then minimize expected cost subject to the 50% approval floor.

For a calibrated default probability p, reject when p × $8,000 exceeds (1 - p) × $900, giving a cost-only cutoff of 10.1%.
I fit Platt scaling and isotonic regression on separate temporal calibration folds, choose by out-of-fold Brier score, and keep isotonic only if its extra flexibility is stable.
I inspect reliability curves and Brier reliability, resolution, and uncertainty components overall and by score band; ROC AUC cannot reveal probability bias.
If the 10.1% cutoff approves less than 50%, I quantify the extra expected loss at the policy-constrained cutoff instead of presenting it as model-optimal.

Why interviewers ask this: This tests calibration, Brier decomposition, cost-based classification, and transparent handling of a policy constraint.

evaluationdecision-making

I would recalibrate pCTR and bid from expected value, because ranking quality alone does not make bids economically correct.

I fit Platt scaling or beta calibration on a recent time split, selecting by Brier score and log loss, then inspect equal-frequency reliability bins by device and placement.
For an impression priced at c, the break-even rule is calibrated pCTR > c / $1.20; a $0.012 impression therefore needs pCTR above 1.0%.
I rank eligible impressions by expected surplus pCTR × $1.20 - c and raise the minimum surplus threshold or scale bids down until projected spend fits the $100,000 budget.
I require lower Brier score, reliability error below 0.2 percentage points in material bins, and no loss in value-weighted PR AUC before an online revenue test.

Why interviewers ask this: This checks whether calibrated probabilities are connected to auction prices, budget allocation, and segment-level reliability.

escalationmonitoring

I would minimize expected harm under the 12,000-review cap, with recall at capacity as the main operational metric.

With calibrated probabilities, review when p × $200 exceeds (1 - p) × $8, which gives a cost-only threshold of 3.85%.
If more than 12,000 items clear 3.85%, I take the top 12,000 by expected harm; if fewer do, I do not fill the queue with negative-value cases.
I report recall@12k, precision@12k, PR AUC, and expected harm per 10,000 uploads; 99.6% accuracy from predicting safe is useless.
I bootstrap by uploader, not item, and require recall floors for high-severity categories so a strong aggregate score cannot hide a dangerous subgroup.

Why interviewers ask this: This tests class-imbalance reasoning, asymmetric costs, clustered data, and capacity-aware thresholding.

cssmonitoring

I would optimize an 80th-quantile demand forecast with business-weighted pinball loss rather than minimize RMSE.

The cost-optimal quantile is $6 / ($6 + $1.50) = 0.80, so the prediction deliberately carries more stock than a median forecast.
I report realized dollar loss, q80 pinball loss, signed unit bias, and service level by SKU-volume band; one aggregate WAPE can hide failures on intermittent items.
The 8-week backtest uses rolling daily origins and only inventory, promotion, and calendar data known before each replenishment cutoff.
I compare gradient boosting with seasonal-naive and Croston-style baselines, and keep the simpler method wherever its dollar loss is statistically indistinguishable.

Why interviewers ask this: This tests whether the candidate can turn asymmetric inventory costs into a quantile objective and validate it on realistic forecast origins.

evaluationretentionchurn

I would choose model B for this campaign, provided the profit difference is statistically and temporally robust.

The break-even churn probability is $10 / (0.20 × $120) = 41.7%, then the 8,000-contact cap selects the highest expected-profit eligible customers.
B wins the decision metric, profit@8k, by $13,000 and has better probability accuracy despite lower ROC AUC; A's stronger global ordering is not the business objective.
I use paired customer-level bootstrap intervals for the $13,000 difference and inspect reliability curves around 30% to 60%, where contact decisions change.
I would also compare isotonic recalibration of A on a separate fold, but ship A only if recalibration raises validated profit@8k above B without unstable bins.

Why interviewers ask this: This tests whether the candidate prefers calibrated decision value at the operating capacity over a higher discrimination metric.

validation

I would use approval-time chronological cohorts and reconstruct every feature as it existed when the lending decision was made.

I would train on approvals from 2021-01-01 through 2023-12-31, validate on 2024-01-01 through 2024-06-30, and lock 2024-07-01 through 2025-06-30 as the test set after all 90-day outcomes mature.
I would remove collections_status, current_days_past_due, repayment totals, charge-off fields, and any bureau refresh received after approval rather than trusting their column names.
I would require each feature timestamp to be at or before approval time and reproduce the join from immutable source history; a feature without historical availability evidence would not enter the model.
I would tune only on validation, open the test once, and report pooled and per-lender results so a lender-mix change cannot hide leakage or weak generalization.

Why interviewers ask this: The interviewer is evaluating whether the candidate can turn an obvious leakage risk into a precise chronological split and an enforceable feature-availability audit.

churn

I would validate forward in time with 30-day purge windows, because a random row split would mix seasons and overlapping outcome windows.

I would train on feature dates from 2022-01-01 through 2024-05-31, purge June 2024, and validate on 2024-07-01 through 2024-11-30.
I would purge December 2024 and lock 2025-01-01 through 2025-11-30 as the test period; December 2025 rows stay out because their 30-day labels are not all mature on 2026-01-31.
I would fit preprocessing only on the training period and score every row using the feature snapshot available on that row's date, without carrying future category frequencies or imputations backward.
I would report each month and each of the six markets separately, then weight them by the expected January and February 2026 traffic mix rather than trusting one aggregate score.

Why interviewers ask this: A strong answer uses calendar-ordered cohorts, purges overlapping label horizons, and matches evaluation to the months the model must predict.

cross-validationvalidation

I would run five-fold stratified grouped cross-validation with patient_id as the indivisible group.

I would assign every image from one patient to exactly one fold, so prior and follow-up studies can never appear on opposite sides of a split.
I would construct five roughly 20% patient folds with seed 20250630, balancing the patient-level label and hospital counts as closely as the group constraint permits.
Each run would train on four folds and predict the fifth, producing one out-of-fold prediction per patient within the five-run budget.
I would aggregate metrics at patient level and report them by hospital as well as overall, because image-level confidence intervals would overstate the effective sample size.

Why interviewers ask this: The interviewer is checking whether the candidate recognizes the patient as the independent unit and can preserve class and hospital coverage under a hard compute limit.

evaluationvalidation

I would reserve one untouched test set and use repeated stratified cross-validation on the remaining applicants to measure split sensitivity.

I would place a deterministic stratified 20% of applicants in the locked test set, balancing default status and the eight acquisition channels, and keep the other 80% for development.
On development data I would run five-fold stratified cross-validation four times with seeds 11, 29, 47, and 83, for 20 fits with different fold assignments.
I would compare fixed candidate settings using mean PR-AUC, its dispersion across the 20 folds, and channel-level recall rather than selecting the highest single fold.
After choosing the setting, I would refit once on all development data and evaluate once on the locked test, using 22 of the 25 available full fits.

Why interviewers ask this: The interviewer is evaluating whether the candidate can use repetition to quantify partition variance without repeatedly consulting the final test set.

cross-validationhyperparametersconfig

I would use patient-grouped nested cross-validation so model selection and performance estimation never use the same held-out patients.

The outer loop would be five-fold StratifiedGroupKFold by patient_id, with label and lab distributions balanced across the five roughly 20% patient folds.
Inside each outer training partition, a four-fold StratifiedGroupKFold would rank all 60 configurations by patient-level PR-AUC while keeping every patient's slides together.
I would refit the inner winner on the full outer training partition and score the untouched outer fold, consuming 1,200 inner fits plus five outer refits.
I would report the distribution of the five outer scores, including critical-subtype sensitivity, then select a configuration by its inner-loop ranks and use one final fit on all 15,600 patients.

Why interviewers ask this: The interviewer is testing whether the candidate can separate hyperparameter selection from unbiased estimation while respecting patient dependence and a precise fit budget.

warehouse

I would split by scoring cutoff and rebuild each feature from records that were genuinely available by that cutoff.

I would train on cutoffs from 2022-01-31 through 2023-12-31, validate on 2024-01-31 through 2024-06-30, and test on 2024-07-31 through 2025-04-30; May and June 2025 stay out because their 60-day labels are immature.
Every join would require event_time at or before the cutoff and valid_from at or before the cutoff with valid_to after it, using append-only history rather than the current corrected row.
I would discard a feature if its historical versions cannot be reconstructed, and I would build rolling aggregates with windows ending at the cutoff rather than at extraction time.
I would fit encoders and imputers on the training cutoffs only, then replay the complete four-hour month-end computation for validation and test and report results by lender.

Why interviewers ask this: A strong answer couples a mature temporal holdout with point-in-time joins that can be reproduced under the actual scoring deadline.

validation

I would evaluate only on eventually mature cohorts while reproducing the seven-day label view available at every weekly training cutoff.

I would train on impressions from 2024-01-01 through 2025-06-30, validate on 2025-07-01 through 2025-09-30, and lock 2025-10-01 through 2025-12-31 as test, whose 30-day outcomes are mature by 2026-01-31.
For each simulated weekly fit, impressions aged 7 to 29 days would remain censored rather than being marked as non-purchases; rows at least 30 days old supply complete labels.
If recent rows are needed, I would estimate conversion-delay probabilities from mature training cohorts and apply a fixed censoring correction learned without validation or test outcomes.
I would score validation and test against their final 30-day labels by country and impression week, so different reporting delays or country mixes cannot create an artificial gain.

Why interviewers ask this: The interviewer is evaluating whether the candidate distinguishes an unobserved future conversion from a negative label and tests the correction on fully matured outcomes.

design

I would hold out sellers by onboarding date and evaluate only their first 14 days, because random interactions would let seller identity and prior engagement leak into cold-start evaluation.

Training would use interactions through 2025-06-30 from sellers onboarded before 2025-07-01, with all vocabularies, popularity priors, and preprocessing fitted on that population.
Validation would contain the first 14 days for sellers first onboarded from 2025-07-01 through 2025-09-30, and it alone would drive feature and threshold choices.
Test would contain the first 14 days for sellers first onboarded from 2025-10-01 through 2025-12-17, leaving enough calendar time for outcomes through 2025-12-31 and verifying the 10,000-purchase minimum before locking the split.
Seller IDs, item interaction histories, and post-onboarding aggregates for validation and test sellers would be unavailable, while listing content and category fields present at onboarding would remain allowed.

Why interviewers ask this: The interviewer is checking whether the holdout matches the stated new-seller use case instead of measuring an easier warm-start problem.

correlationdeploymentvalidation

I would make R16 through R18 a locked spatial test and tune with four contiguous regional folds inside R01 through R15.

No row from R16, R17, or R18 would enter training, preprocessing, or threshold selection, and I would remove training cells within 25 km of their borders to limit spatial bleed.
The four validation folds would be contiguous blocks R01-R04, R05-R08, R09-R12, and R13-R15, with each run training on the other three blocks across the full 2010-2024 period.
Weather normalization, missing-value rules, and spatial aggregates would be fitted within each training side, and neighbor features could not cross the held-out block boundary.
I would select the fixed setting from the four validation results, refit on eligible R01-R15 cells, and report R16, R17, and R18 separately as well as pooled.

Why interviewers ask this: A strong answer makes geographic independence explicit, includes a correlation buffer, and evaluates the exact unseen regions named in the deployment requirement.

I would run six expanding-window forecast origins and preserve exactly the information available at each origin.

The origins would be 2024-07-01, 2024-10-01, 2025-01-01, 2025-04-01, 2025-07-01, and 2025-10-01, each scored on the following 28 calendar days.
For each origin, training would start on 2019-01-01 and include only examples whose full 28-day target window ends before that origin, preventing target overlap with the forecast window.
Promotion values would be used as known for days 1 through 14 and masked or replaced by the planning-time fallback for days 15 through 28, never filled from realized promotions.
I would use the first five origins for the final setting decision and open the October 2025 origin once as the locked result, reporting weighted error by store, SKU, and zero-demand cohort.

Why interviewers ask this: The interviewer is evaluating whether the candidate can specify reproducible forecast origins, honor target and feature availability, and reserve the latest period for a final decision.

Locked questions

21
RideNow has 2 billion trip and app events and needs 7-day and 30-day rider features for a cancellation model trained on 120 million orders. In Spark, how would you choose between window functions, range joins, and pre-aggregated snapshots while preserving point-in-time correctness?
joinswindow-functionsaggregation
22
ShopGrid has 80 million training rows, 12 million seller IDs, and a conversion target with a 21-day feedback delay. Would you use one-hot encoding, hashing, frequency encoding, or target encoding for seller ID, and how would you prevent leakage across 6 monthly validation folds?
feedbackvalidation
23
SupportFlow must classify 600 million support messages using a 384-dimensional sentence encoder, but storing float32 embeddings would consume about 922 GB before replicas. Would you precompute embeddings, compute them during each training run, or retain sparse text features instead?
nlpreplication
24
AdAtlas trains a click model on 3 billion examples with 8 million possible query tokens and 250 dense numeric features. A dense matrix cannot fit in the 1.5 TB Spark training cluster, so what sparse representation and model path would you choose?
queriestokens
25
CarePath predicts 30-day readmission from 90 lab features across 45 million visits; 38 percent of lab values are missing, and one hospital orders a key test for 82 percent of severe patients but only 24 percent of others. How would you represent missingness without leaking post-admission decisions?
26
FinCredit has 40 million applications and 50,000 candidate features generated by vendors, aggregates, and text pipelines, but its weekly training budget is 6 hours. How would you reduce the set to roughly 2,000 features without selecting on the final test period?
procurementaggregationci-cd
27
StreamBox needs 1-hour, 1-day, 7-day, and 28-day viewing aggregates for 140 million users from 2 billion daily events. Would you scan raw history for every Spark run or maintain composable aggregate tables, and at what grain?
aggregation
28
FreshCart has 200 base features and suspects that user-by-store, user-by-category, and price-by-promotion interactions drive substitutions across 70 million baskets. Exhaustive pairwise crosses create 19,900 candidates before categorical expansion; which interactions would you materialize?
29
LakeCart builds 300 features daily for 100 million customers from 4 TB of new events; recomputing 30 days scans about 120 TB per run, while materializing all daily customer features adds 18 TB per month. How would you decide what to compute once, cache temporarily, or persist?
caching
30
MegaPlay aggregates 2 billion daily events in Spark, but 6 percent use the anonymous user ID and the largest real account has 14 million events. How would you design user-level feature aggregation so these keys do not dominate a 4,000-task shuffle?
aggregationdesign
31
You are modeling 90-day default from 1.2 million loan applications with a 6% default rate. Regularized logistic regression reaches 0.41 PR-AUC and XGBoost reaches 0.44, while policy requires stable reason codes, p95 scoring below 20 ms, and a monthly training budget of $2,000. Which model do you ship?
boostingevaluationregression
32
An e-commerce return model has 8 million labeled orders, a 3% positive rate, 220 tabular features, and no images or text. XGBoost achieves 0.36 PR-AUC, while a four-layer neural network achieves 0.365 after 48 GPU-hours; serving must stay below 10 ms p95 and monthly GPU spend below $3,000. Which model family do you choose for the next release?
boostingneural-netsevaluation
33
For 250,000 cancer-screening records with a 1.5% positive rate, logistic regression has 0.86 AUROC and XGBoost has 0.90, but both must achieve at least 92% sensitivity. A missed case is estimated at $20,000, a specialist review costs $80, regulators require an auditable rationale, and p95 latency must remain below 50 ms. How do you resolve interpretability versus accuracy?
boostingregressionlatency
34
An XGBoost fraud model trained on 600,000 transactions with a 0.7% fraud rate reports 0.92 train PR-AUC and 0.48 validation PR-AUC after 800 trees; the previous model reached 0.51 validation PR-AUC. Retraining is limited to $300 per week and serving to 25 ms p95. How do you diagnose and correct the overfitting?
overfittingboostingevaluation
35
You have 10 million card transactions with a 0.08% fraud rate, a target of at least 70% recall at 20% precision, a $900 cost per missed fraud, a $3 investigation cost per alert, and a 15 ms p95 scoring limit. How would you handle the imbalance while training the model?
transactionsalerting
36
A sparse logistic model for account takeover uses 4 million sessions, 120 million hashed features, and a 0.25% positive rate. Full-data training fits in 32 GB, the target is 0.30 PR-AUC, false negatives cost $500, false positives cost $1, and inference must stay below 8 ms p95. Would you use class weights or negative downsampling?
evaluationsessions
37
A 3-million-parameter churn network is trained on 180,000 customers with a 12% churn rate. Train log loss falls to 0.18 while validation log loss rises from 0.31 to 0.44 after epoch 12; the release target is validation AUROC 0.82, GPU time is capped at 12 hours per week, and scoring must stay below 5 ms p95. What regularization plan would you test?
regularizationvalidationchurn
38
You must tune an XGBoost conversion model on 3 million sessions with a 2% positive rate. The acceptance target is at least 0.24 validation PR-AUC, the search budget is 120 CPU-hours, a false positive costs $0.20, a false negative costs $8, and the final model must score within 12 ms p95. How do you select hyperparameters?
boostingevaluationhyperparameters
39
An insurance XGBoost model is trained on 900,000 policies with a 4% claim rate and reaches 0.89 AUROC, but policies assigned a 10% claim probability realize only 6%. Its Brier score is 0.061 versus 0.054 for logistic regression, a one-point probability bias costs about $150,000 in annual pricing error, calibration may add at most 2 ms to a 20 ms p95 budget, and 120,000 recent labels are available. How do you calibrate it?
boostingregressionprobability
40
A content-safety model was evaluated on 2 million labeled posts with a 0.4% violation rate and serves 5 million posts per day. Scores are calibrated, a missed violation costs $40, a false removal costs $2, manual review is capped at 60,000 posts per day, the target is expected cost below $0.12 per post with at least 85% recall, and model scoring uses 18 ms of a 25 ms p95 budget. How do you choose the decision thresholds?
decision-making
41
A signup experiment has a 10% baseline conversion rate, and the smallest worthwhile lift is 1 percentage point. With 80% power, a two-sided 5% alpha, 4,000 eligible users per day, and 10% expected loss before analysis, what sample and duration would you choose?
experimentsconversion
42
You are testing average order value, currently $48 with a $30 standard deviation, and care about a $2 increase. At 80% power and two-sided 5% alpha, how many observations do you need, and what would you do about the metric's heavy tail?
dispersionmonitoringtesting
43
A store-training treatment must be assigned by store because employees share the new process. An individual-level calculation says 2,500 customers per arm, stores average 120 customers, and the intracluster correlation is 0.04. How would you size and randomize the experiment?
experimentsconcurrencycorrelation
44
A delivery-pricing test creates marketplace interference, so drivers and orders cannot be independently randomized. You have 12 cities and can run for 28 days. Design a switchback test with concrete time blocks, washout, and an analysis unit.
designpricing
45
A fixed-horizon revenue experiment needs 20,000 users per arm, but a pre-experiment revenue covariate has correlation 0.60 with the outcome. What gain do you expect from CUPED, and what conditions must hold before you reduce the sample?
variance-reductionexperimentscorrelation
46
You want to compare three checkout variants with one control. Baseline conversion is 8%, the MDE is 0.8 percentage points, power is 80%, and family-wise alpha is 5%. How would you allocate traffic and handle multiplicity?
47
A team plans a maximum sample of 60,000 users but wants to inspect significance after every day and stop at p below 0.05. Replace that plan with a concrete sequential design.
design
48
A supposedly 50/50 experiment assigns 51,200 users to treatment and 48,800 to control out of 100,000. Calculate the SRM check and decide whether you would read the treatment effect.
experiments
49
A recommendation redesign lifts clicks by 4% in days 1 to 3 but only 0.8% in days 4 to 14, and the test began five days before Black Friday. Would you ship, extend, or stop, and what analysis would support that decision?
50
A product has 12,000 eligible users per week, baseline conversion is 3%, and the experiment can run for at most six weeks. With equal allocation, 80% power, and two-sided 5% alpha, what MDE is feasible, and what if the business only values a 0.2 percentage-point lift?
experiments
51
Thirty minutes after a recommendation model release, click-through rate drops 14%, while traffic and p95 latency stay within 2% of baseline. What incident decision do you make and how do you contain it?
incidentslatency
52
A credit model's income feature reaches PSI 0.34, above the 0.20 alert threshold, but on the latest mature seven-day cohort AUC is 0.81 versus 0.812 baseline and calibration error remains 0.03. New labels arrive with a 21-day delay. Do you roll back, retrain, or monitor, and what containment do you apply?
monitoringrollbackalerting
53
A fraud model approves 18% more transactions than usual, but chargeback labels arrive 14 days later; meanwhile manual-review rejection rises from 6% to 15%. How do you contain the incident without waiting two weeks?
incidentstransactions
54
An operator lowers a fraud threshold from 0.72 to 0.58, blocked payments rise 31%, and confirmed fraud caught rises only 4% over six hours. What do you change first, and how do you decide whether the model also needs intervention?
55
A loan model's income-feature null rate jumps from 2% to 27%, approvals fall 16%, and default labels will not mature for 30 days. What is your first-hour containment plan?
fundamentals
56
A challenger model has 7% higher offline PR-AUC, but at 10% production traffic it reduces conversion by 4.5% and raises complaints by 22% over three days. Do you promote it, tune it, or stop it?
evaluation
57
After a weather-code taxonomy update, an ETA model's weather feature reaches PSI 0.29, but MAE moves only from 6.2 to 6.3 minutes and p95 error stays at 18 minutes across 1.5 million deliveries. How do you decide whether this drift is harmless?
iac
58
A merchant-risk model's global AUC falls only 1%, but recall for new merchants drops 18% and they represent 12% of traffic. What containment and model decision do you make?
evaluation
59
During a nationwide transport strike, a demand forecast's MAPE rises from 11% to 23%; replay shows the previous champion would score 22% on the same hours. Do you roll back or retrain, and how do you contain planning errors today?
rollback
60
A retrained risk model has 9% higher offline AUC, but on a 20% canary its calibration error worsens from 0.04 to 0.13 and approval rate falls 11%; outcome labels take 30 days to mature. Do you roll back, recalibrate, or retrain?
evaluationrollbackdeployment-strategies
61
At 09:20, purchase_count_30d became zero for 93% of requests instead of 4%, and the ranking model serves 38% of traffic. How do you lead the incident?
incidents
62
A new model has offline AUC 0.86, but production precision falls 19% and sampled parity checks show different features on 7.8% of requests. What do you do?
evaluation
63
Three days after launching a chargeback model, you discover that dispute_status was updated up to 14 days after the prediction timestamp and helped produce an offline AUC of 0.94. How do you respond?
evaluation
64
A feature-store join used the latest customer tier instead of the tier at decision time, affecting 11.6% of 2.3 million decisions over six days. How do you handle it?
soft-skillsjoins
65
After a producer release, 41% of merchant_category values map to unknown instead of 2%, authorization rate falls 9%, and a payment-routing model has already scored 68,000 transactions. What is your incident plan?
incidentstransactionsauth
66
After a producer release, package_weight changes from kilograms to grams, moving the median from 1.8 to 1,800 for 18% of traffic. How do you contain and repair the issue?
67
A recommendation release serves query embeddings from encoder v8 against a catalog index built with v7, and recall@20 drops 31%. What do you do?
indexesqueriesnlp
68
Training uses tokenizer v3 with Unicode normalization, but serving still uses v2; 14% of non-ASCII requests produce different tensors and moderation F1 falls 12 points. How do you lead the fix?
normalizationtokens
69
An ETL retry duplicates 6.4 million events, inflating 8.7% of training rows and live order_count features for 180,000 users. What is your response?
resilienceetl
70
Three of 12 feature-pipeline shards stop advancing, leaving 27% of entities a median 47 minutes stale while prediction endpoints still return 200. How do you manage the partial outage?
shardingendpointsci-cd
71
A checkout experiment reaches 240,000 users and shows a -3.8% relative lift in completed purchases with p = 0.012; payment errors are also up from 1.6% to 2.1%. How do you interpret the result, and what do you decide today?
experiments
72
A pricing-page test increases the preregistered primary metric, paid starts, by 2.4% with p = 0.018, but 14-day refund rate rises from 4.9% to 5.8%, a relative increase of 18.4% with p = 0.031, and margin per visitor falls 1.2%. What is your interpretation and launch decision?
pricingmonitoringcss
73
A recommendation test ends early when a partner campaign stops, after 90,000 of the planned 200,000 users. Revenue per user is +1.1% with p = 0.28 and a 95% confidence interval from -0.9% to +3.1%, while all guardrails are neutral. How do you interpret the null and decide what runs next?
confidence-intervalsfundamentalsguardrails
74
A new notification format produces a +8.5% click lift on days 1 to 3, +1.2% on days 4 to 14, and +2.6% overall with p = 0.040; the treatment-by-time interaction has p = 0.009. Unsubscribes are flat. How do you interpret this pattern and decide on rollout?
75
Three days after launch, an experiment expected to split 50/50 has 518,400 treatment users and 481,600 control users, giving an SRM p-value below 0.00000001; the primary metric shows +4.1% with p = 0.020. What do you conclude and do next?
experimentsmonitoringhypothesis-testing
76
An onboarding test shows +1.6% activation overall with p = 0.040, but new-to-category users improve 6.4% with p = 0.003 while experienced users decline 2.8% with p = 0.010; the treatment-by-segment interaction has p = 0.002. What is your interpretation and release decision?
activationonboarding
77
A mobile checkout test reports a +12.0% purchase lift with p < 0.001, but an audit finds that a retry path emits duplicate purchase events for 7.4% of treatment orders versus 0.3% of control orders. Finance shows no corresponding increase in captured payments. How do you interpret and resolve this?
resilience
78
A search-ranking test raises result CTR by 7.8% with p = 0.001 and add-to-cart rate by 3.2% with p = 0.020, but purchases fall 2.4% with p = 0.043 and margin per search falls 3.0% with p = 0.028. Latency is unchanged. How do you reconcile the metrics and decide?
csslatencymonitoring
79
A promotions experiment runs from December 20 through January 10 and shows +4.8% revenue per visitor overall with p = 0.030. The lift is +12.0% during December 20 to 26 with p = 0.004 but -3.1% from January 2 to 10 with p = 0.040, and 70% of observations came from the holiday week. What do you infer and launch?
experiments
80
A retention test misses its preregistered primary metric at -1.7% with p = 0.21. Among 20 post-hoc country-platform segments, iOS users in Canada show +14.0% with p = 0.030, and a stakeholder asks you to present only that segment and launch there. What do you say and decide?
retentioncommunicationmonitoring
81
After 12,400 fraud decisions, VIP customers are 3% of volume but account for 28% of false positives, and the stakeholder says the model is wrong. How do you respond?
stakeholder-managementcommunication
82
A threshold of 0.42 catches 89% of fraud but creates 1,600 reviews per day against an operations capacity of 900. The team refuses your recommendation. What do you do?
capacity
83
Across 20,000 applicants, a hiring model selects 18% of men and 11% of women, with qualified-candidate recall of 82% versus 61%, while overall accuracy matches the incumbent. What do you do?
84
A credit model approves 79% of urban applicants but only 54% of rural applicants. How would you decide whether to intervene?
85
A retention model sends offers to 32% of customers under 55 but only 14% of customers aged 55 and older. What is your response?
retention
86
A medical triage model has a 6% overall false-negative rate but a 19% rate for women aged 65 and older. What action do you take?
87
On 60,000 loan applications, postcode, branch distance, and property value predict an audited protected group with AUC 0.88, and matched applicants still show a 12-point approval gap. How do you investigate and mitigate proxy discrimination?
evaluationproxy
88
In 8.2 million sensor rows, model explanations say low equipment temperature raises failure risk, but 74% of high-risk alerts come from firmware v3 and reliability engineers say the direction contradicts physics. What do you do?
alerting
89
An executive asks for one launch number after a 21-day recommendation test on 18 million impressions shows +$1.24 gross profit per 1,000 impressions, with a 95% interval of +$0.40 to +$2.05, while complaints rise 0.3 points. What do you present?
90
A fraud model improves offline AUC from 0.86 to 0.94, but recovered losses do not change and analysts override 64% of its alerts. How do you respond?
evaluationalerting
91
A junior data scientist trains a churn model on 1.8 million subscription rows and reports 0.99 validation AUC two days before the review. You notice that cancellation_date and refund_status were included as features, while the model must score users 14 days before cancellation. If this result determines whether a six-week retention pilot launches, what do you do?
retentionchurnvalidation
92
A mid-level data scientist is building a fraud classifier from 420,000 transactions with only 1,900 confirmed fraud cases. After 60 Optuna trials, training PR AUC rose from 0.71 to 0.94 but time-based validation PR AUC fell from 0.69 to 0.61, and the model review is in three days. What do you do?
evaluationtransactionsvalidation
93
A colleague wants to replace a logistic credit-risk model with a stacked ensemble on 3.2 million loan records. Offline default-detection AUC improves from 0.842 to 0.845, but p95 scoring latency rises from 18 ms to 140 ms and monthly infrastructure cost is estimated to increase by $11,000; the architecture decision is due Friday. What do you do?
evaluationensemblesarchitecture
94
At 10:00 Tuesday, product asks for an analysis of a checkout redesign using 12 million events, with a recommendation for Wednesday's 15:00 launch meeting. Event exposure is reliable, but the revenue table is missing about 8% of mobile purchases and a full repair will take four days; the launch affects roughly $600,000 in weekly revenue. What do you do?
95
Your team is asked to launch a no-show risk model trained on 900,000 hospital appointments in two weeks. An audit shows that at the proposed threshold it flags 34% of patients from one low-income district versus 12% overall, and flagged patients may lose access to preferred appointment slots; the measured recall gain is 6 percentage points. What do you do?
96
During review of a board deck due tomorrow, you see the claim that a recommendation model increased 30-day retention by 9%. The analysis covers 46,000 users, but assignment was based on device support rather than randomization, baseline retention differed by 7 points, and the reported p-value ignores that imbalance. What do you do?
hypothesis-testingretention
97
A strong mid-level data scientist needs an ownership example for promotion in six months. The team has a stale demand-forecast pipeline for 1,200 stores with 18% weekly MAPE, about $2 million in annual stockout cost, and no named owner; two engineers can each contribute one day per week. How do you mentor this project?
mentoringownershipci-cd
98
Six hours before a holiday campaign launch to 4 million customers, marketing wants to lower the propensity threshold from 0.62 to 0.48 to fill all 700,000 message slots. You disagree because last month's holdout suggests the extra 180,000 recipients have negative expected margin after a $0.09 send cost, but the campaign owner says the inventory is already paid for. What do you do?
designconflictcss
99
After four months, a delivery-time model trained on 28 million orders fails its pilot: median error improves by 6%, but p95 error worsens by 22%, customer complaints rise by 4%, and operations will not expand beyond 5% of traffic. You led the project and the quarterly review is next week. What do you do?
100
A junior data scientist owns monitoring for an ETA model serving 9 million predictions per day. On Monday they notice p95 absolute error rose 11% after Friday's release, but they waited seven hours to report it because mean MAE stayed green; the next release window closes in 24 hours. What do you do?
mlopsmonitoring

Questions

1A subscription service has 500,000 active customers, 6% monthly churn, a $90 retained margin, and a $12 retention offer. An experiment says the offer prevents 25% of churn, but operations can contact only 40,000 customers per month. Which metric and decision threshold do you use?

2A payment processor scores 10 million transactions per day; confirmed fraud is 0.15%, a missed fraud costs $900, a manual review costs $6, and reviewers can inspect 5,000 transactions daily. How would you choose the metric and review threshold?

4An e-commerce search team has an offline set of 2 million queries, a 1.2% purchase rate, an average $65 order margin, and only the top 10 results are visible without scrolling. The reranker must stay below 80 ms p95 latency. Which ranking metric and launch bar do you set?

7An ad model is evaluated on 5 million recent impressions with a 0.8% click rate. A click is worth $1.20, auction prices range from $0.002 to $0.03 per impression, and daily spend cannot exceed $100,000. Offline AUC improved, but reliability plots show underprediction above 2% CTR. What do you do?

8A content platform reviews 300,000 uploads per day; 0.4% contain severe policy violations, a missed violation is valued at $200 of harm, a false escalation costs $8, and moderators can review 12,000 items daily. Which metric and threshold would you use?

9A retailer forecasts 14 million SKU-store-day rows with 35% zero demand. Underforecasting one unit loses $6 of margin, overforecasting one unit costs $1.50 in markdowns, replenishment runs daily, and the final backtest covers 8 weeks. Which forecast and evaluation metric would you use?

Locked questions