MLOps Engineer interview questions

I would join every label to feature values that were available at the original prediction time, not when the training job runs.

• Feast historical retrieval should match entity keys and event timestamps, with the label window starting 14 days after prediction so future transactions cannot leak into a row.
• I would retain event time and ingestion time, accept data up to a 48-hour watermark, and version corrections to already published partitions.
• A test would replay 10,000 entity-timestamp pairs in BigQuery and fail if any selected feature timestamp is later than its prediction timestamp.

I would require a versioned FeatureView contract that makes semantics, ownership, freshness, and compatibility testable.

• The definition must name the entity key, type, unit, null policy, default, owner, source, and a freshness target such as 15 minutes.
• Type or semantic changes create a new version with a 30-day dual-read window; additive metadata changes can stay backward compatible.
• CI compares 10,000 representative rows from the warehouse and Redis, then blocks publication on schema mismatches or values outside the declared tolerance.

I would keep history in the warehouse and materialize only current serving values into a partitioned online store.

• BigQuery or Snowflake holds the 50 TB history and runs point-in-time joins without forcing the online tier to retain old versions.
• Kafka and Flink materialize updates into Redis Cluster sized for 130,000 lookups per second, leaving 30% headroom above peak.
• Feast binds both paths to one definition; five-minute freshness checks and checksum samples over 1 million keys catch incomplete materialization.

I would measure freshness at read time from the source event timestamp and make stale behavior explicit in the serving contract.

• Flink writes the event timestamp with each Redis value, and the client records source lag, processing lag, and served age separately.
• Prometheus evaluates the share of reads older than 60 seconds over rolling 10-minute windows and pages only after two consecutive violations.
• The model either uses a versioned fallback value or rejects the prediction after 120 seconds; silently serving an unmarked stale value is not allowed.

I would isolate historical computation from online publication and promote only the latest valid value after validation.

• Spark writes month-sized offline partitions to shadow tables with pinned source snapshots and resumable partition IDs.
• Backfill compute uses a separate queue, while Redis writes are capped at 5,000 per second against a measured 20,000 per second spare budget.
• A cutover selects the newest event per entity and uses timestamp guards, so an old backfill row can never overwrite a newer streaming value.

I would publish a new feature version and migrate each model independently instead of changing the existing value in place.

• Feast exposes average_order_value_v2 while v1 remains available for at least 30 days with separate lineage and ownership.
• Both versions are materialized to Redis, and Evidently compares seven days of coverage, null rate, and PSI with a review threshold of 0.10.
• The three models move through separate Argo CD releases; v1 is deleted only after registry lineage shows zero training and serving consumers.

I would reduce network round trips and precompute request-ready groups rather than issue 40 independent reads.

• Feast feature services define the exact 40-feature vector, and Redis stores co-read values under two or three entity keys instead of 12 remote calls.
• The budget reserves 3 ms for the client and network, 8 ms for Redis, and 4 ms of tail margin, with a hard 12 ms client deadline.
• Load tests at 60,000 RPS track Redis command latency, payload size, connection-pool waits, and hot-key rate before accepting the layout.

I would use model-approved fallbacks with explicit age and missingness, not convert every failed read into zero.

• The client keeps a bounded local cache for low-cardinality features and accepts values no older than 10 minutes only where the feature contract permits it.
• Missing online values map to versioned defaults from the training dataset and set missingness indicators the model saw during evaluation.
• A 20 ms deadline and circuit breaker stop retry storms; the degraded prediction rate is measured separately and must stay below 0.5% monthly.

I would deploy 116 replicas because 89 cover the measured peak and a 30% capacity reserve raises that to 116.

• The end-to-end budget allocates 8 ms to routing, 32 ms to inference, and 10 ms to queueing and tail variation.
• Replicas are spread across three zones with topology constraints, and each zone can lose 10 replicas without exceeding safe concurrency.
• Admission control rejects work when queue age passes 8 ms, because adding an unbounded queue would preserve throughput while breaking p99.

I would sweep batch size and queue delay against production-shaped traffic and choose the fastest setting that stays inside 35 ms p99.

• Triton Model Analyzer tests batches 4, 8, 16, and 32 with queue delays from 0.5 to 4 ms, recording queue time, execution time, and achieved batch distribution.
• Inputs are bucketed by shape, because padding a mixed batch to the largest image can erase the throughput gain and raise memory use.
• I keep 20% throughput headroom and reject a setting if queueing exceeds 8 ms at 7,200 predictions per second, even when average latency looks good.

I would tune token scheduling and tensor parallelism from the real prompt and output distributions, not a fixed request batch size.

• Benchmark 4 and 8 H100 tensor-parallel layouts with production p50 and p95 prompt lengths, measuring TTFT, time per output token, KV-cache occupancy, and tokens per second.
• Set max_num_batched_tokens so prefill cannot starve decode, and separate interactive requests from batch traffic with distinct queues and concurrency limits.
• Keep KV-cache occupancy below 90% at peak and shed low-priority work when oldest queue age reaches 1.2 seconds, leaving 600 ms for prefill and network tail.

I would isolate concurrency, queues, and scaling by model while keeping a shared routing contract.

• Popular models get dedicated warm Triton instance groups; rare models may share a pool only if their combined load and memory fit a tested limit.
• The router applies per-model deadlines, concurrency caps, and bounded queues, then routes by immutable model version rather than a mutable latest tag.
• Dashboards split RPS, p99, queue age, GPU memory, and rejection rate by model, and a noisy model loses its own capacity before consuming another model's reserve.

I would scale on queue demand with scheduled warm capacity that covers the four-minute startup gap.

• Keep two warm GPUs for 300 RPS and add one replica per measured 150 RPS when oldest queue age exceeds 20 ms for two minutes.
• A forecast raises the minimum 10 minutes before the daily peak; scale-down waits 15 minutes to avoid unloading a model that will immediately return.
• Cap the fleet at 20 GPUs and shed low-priority traffic above 2,700 RPS, while comparing p99 and cost per million requests with a fixed-capacity baseline.

I would run complete regional serving stacks and keep model and feature reads inside the selected region.

• Each region normally handles about 13,300 RPS but is sized for 24,000, so the remaining two can absorb all traffic after one region is removed.
• The 60 ms budget assigns 15 ms to global routing and network, 15 ms to features, 20 ms to inference, and 10 ms to serialization and tail margin.
• The global load balancer removes an unhealthy region within 10 seconds, and quarterly load tests verify 40,000 RPS without synchronous cross-region calls.

I would use deterministic cohorts and increase traffic only after both service and model metrics pass a fixed observation window.

• Route 1%, 5%, 20%, and 50% for 30 minutes each, keeping users pinned by account ID so comparisons are not contaminated by version switching.
• Roll back if p99 rises by more than 10 ms, errors exceed baseline by 0.2 percentage points, or the primary ranking metric drops more than 1%.
• Argo Rollouts moves an immutable model and image digest, while MLflow stores the exact baseline, candidate, thresholds, and decision as release evidence.

I would mirror eligible requests asynchronously and isolate shadow capacity from the response-serving path.

• The gateway publishes sampled request references to Kafka after the production response path, and a separate consumer calls the candidate with side effects disabled.
• Correlation IDs join baseline and candidate outputs for 3,000 RPS of paired analysis, while sensitive fields are removed before the mirror topic.
• Shadow workers have their own GPU quota and drop work when lag exceeds five minutes, because replay completeness is less important than production p99.

I would stage immutable weights near the GPU nodes and keep readiness false until the runtime completes a real warm-up.

• A DaemonSet or image pre-puller caches the model digest on local NVMe, reducing the 90-second remote transfer to a measured load from local disk.
• Triton loads the model explicitly and runs 20 representative warm-up requests before the readiness probe admits traffic.
• Keep at least one spare warm node per zone; this costs idle capacity but is required because autoscaling cannot beat a 25-second target from a cold remote download.

I would use RawDeployment for the 70 ms endpoint unless measured Knative queueing and cold starts fit the budget.

• RawDeployment gives direct control of pods, HPA signals, disruption budgets, and warm replicas, which suits steady high-volume GPU traffic.
• Serverless mode is useful for sparse CPU models that tolerate scale-to-zero, but a 20-second model cold start cannot satisfy a synchronous 70 ms request.
• I would load-test both at the same 1,000 RPS profile and compare proxy overhead, p99, scale-up time, and cost before standardizing the runtime class.

I would declare both as tested deployment targets and route by queue headroom, not use CPU as an unverified emergency fallback.

• GPU handles the steady peak where batching lowers unit cost; CPU absorbs low-volume periods if its full path remains below 100 ms p99.
• The router sends work only to pools with queue age below 15 ms and preserves 20 ms for feature retrieval and network overhead.
• Compare dollars per million successful predictions at 100, 1,000, and 10,000 RPS, including idle GPU cost, before setting the crossover point.

It is only correct if the $18,000 includes all attributable work and the denominator is defined as successful production predictions.

• Include GPUs, CPUs, gateways, storage, network, licenses, and telemetry, and report idle or unattributed spend instead of silently spreading it.
• Keep failed, retried, shadow, and warm-up work in the numerator but outside the successful-prediction denominator, with separate rates for diagnosis.
• Also report cost per token or item when request sizes vary, because a cheap average can hide one cohort consuming most accelerator time.