LLM Engineer interview questions

I would separate a policy-aware gateway, token-based scheduler, model pools, and versioned capability registry so routing and capacity remain explicit.

• The gateway enforces trust tier, region, context limit, tools, schema, quotas, and deadlines before selecting any hosted or self-hosted route.
• Hot models get resident vLLM or TGI pools sized from prompt and output distributions, while cold variants use isolated pools with declared cold-start behavior.
• Load tests at 26,000 requests per minute verify 30% headroom, p99 TTFT, output tokens per second, quality, and cost for every route.

I would route by data eligibility and measured task quality first, then optimize cost inside each eligible pool.

• Regulated prompts stay on an approved private deployment with pinned weights, regional storage, and audited retention; nonregulated traffic may use hosted models.
• A common request contract preserves messages, tools, schemas, deadlines, and usage, while provider-specific adapters expose features without claiming model equivalence.
• Shadow evaluation compares quality, p95, effective cost per successful task, and capacity before any portion of the 200 million daily tokens changes route.

I would use constrained decoding against a versioned JSON Schema and keep semantic validation outside the model.

• Outlines, Guidance, or a provider-native grammar limits token choices for syntax, while enums, ranges, and cross-field rules run in deterministic code.
• Repair retries are capped and receive exact validation failures; truncated, contradictory, or policy-invalid outputs take a separate fallback path.
• The release gate covers exact schema validity above 99.95%, field accuracy, retry rate, p99, and tokens on production-shaped payloads.

I would operate it as a resumable token-aware pipeline and prove required throughput and cost from a representative pilot.

• Immutable manifests and deterministic record IDs support stage checkpoints, idempotent writes, classified retries, and resume without duplicate charges.
• Queues group records by model, prompt length, and output bound, applying provider batch APIs or self-hosted continuous batching under quota and GPU backpressure.
• Hourly forecasts track accepted records per second, quality-pass rate, remaining tokens, completion probability, and projected spend against 18 hours and $120,000.

I would keep shared base replicas resident and dynamically load versioned adapters through a bounded cache keyed by tenant and base revision.

• The registry verifies tokenizer, target modules, rank, base hash, evaluation, and tenant authorization before an adapter can load.
• Hot adapters are replicated from measured demand, while cold adapters accept a declared load delay and eviction uses memory pressure plus recent frequency.
• Requests with different adapters batch only when the engine preserves adapter isolation, and no KV or adapter state may cross tenant boundaries.

I would make hard eligibility deterministic and use a calibrated difficulty policy to select the cheapest model meeting each task's quality floor.

• Context, language, modality, tools, data policy, and latency exclude incapable routes before a learned router considers cost.
• Randomized holdouts and shadow labels estimate false cheap-route acceptance, escalation, p99, and total success without router selection bias.
• The full routed policy, not each model alone, must meet quality and cost targets under the production task mixture.

I would start with LoRA because the dataset and 36-hour budget do not justify optimizer states and gradients for all 70B parameters without evidence of a quality ceiling.

• A pilot sweeps rank 16, 32, and 64 plus attention-only versus attention-and-MLP targets on a fixed held-out set.
• BF16 base weights remain frozen, while adapter checkpoints, optimizer state, tokenizer, data manifest, and exact base revision are versioned.
• Full fine-tuning is considered only if LoRA materially misses the task floor and a measured throughput and memory plan fits eight H100s.

I would load the frozen base in 4-bit NF4 with double quantization, train BF16 LoRA adapters, and spend memory on activations and sequence throughput.

• Start with rank 32, alpha 64, dropout near 0.05, and target attention plus selected MLP projections, then ablate narrower targets.
• Gradient checkpointing, packed sequences, gradient accumulation, and a distributed sampler keep four GPUs productive without exceeding memory.
• Compare QLoRA with BF16 LoRA on held-out quality, training tokens per second, peak memory, and merge or serving compatibility before accepting its savings.

I would choose rank 16 unless the 1.1-point gain at rank 64 clears a predefined critical-slice requirement.

• Adapter parameters and memory scale roughly with rank, so rank 64 makes storage, network loading, and resident adapter cache about four times heavier.
• Compare the ranks by task slice, forgetting, latency, batch density, and cold-load time rather than aggregate success alone.
• If one high-value task needs rank 64, I would keep that adapter exceptional instead of charging all 100 tenants the same cost.

I would choose mixture weights from target product importance and gradient value, not copy the raw 70:20:10 collection ratio.

• Deduplicate first, cap repeated support templates, and build per-domain validation sets with rare and high-risk cases.
• Run short mixture pilots such as 50:30:20 and temperature-smoothed sampling, measuring task gains and interference after equal training tokens.
• The final manifest records source counts, sampling weights, licenses, consent, and effective examples seen so the run is reproducible.

I would remove exact and semantic duplicates and quarantine benchmark contamination before any train-validation split.

• Normalize stable fields, hash exact records, and use MinHash or embedding clusters with source-aware thresholds for near-duplicate groups.
• Match benchmark prompts, references, and paraphrases against all sources, retaining provenance so suspected contamination can be excluded and audited.
• Split by document, customer, conversation, and time boundary after grouping, then report removed counts and contamination rates by source.

I would pack multiple examples into each sequence while preserving loss and boundary semantics.

• Insert the correct EOS or separator, mask padding and any prompt tokens excluded from loss, and ensure one example cannot leak into another through labels.
• Bucket by length before packing so long examples do not create new padding, and keep deterministic packing in the dataset manifest.
• Measure nonpadding-token ratio, examples per step, tokens per second, convergence, and held-out quality rather than GPU utilization alone.

I would calibrate the schedule with a short range-finding run and define it in tokens rather than examples or wall-clock time.

• Start conservatively for LoRA, compare candidate peak rates such as 1e-4 and 2e-4, and warm up roughly 1% to 3% of planned training tokens.
• Track train and validation loss by domain, gradient norm, update norm, throughput, and task evals every fixed token interval.
• Stop from held-out task saturation or regression, not merely completion of an arbitrary epoch over duplicated data.

I would clean and slice the pairs, pin a reference model, and tune DPO strength against preference gain and capability regression.

• Agreement analysis separates ambiguous pairs, systematic reviewer differences, and label errors; uncertain pairs are excluded or weighted deliberately.
• Start with a small beta sweep such as 0.05, 0.1, and 0.2 plus length controls, then inspect chosen-rejected margins by task.
• Release evaluation covers blinded pairwise win rate, factuality, safety, verbosity, KL movement, and base benchmarks on a sealed holdout.

I would pilot PPO only if the simulator reward is robust and online policy exploration can address failures that static pairs do not cover.

• Validate reward precision on adversarial trajectories, partial success, invalid tools, and shortcut behavior before letting the policy optimize against it.
• The PPO stack needs rollout workers, reference and value models, advantage estimation, KL control, and monitoring for reward growth without task improvement.
• Continue only if held-out task success rises materially above 58% and the gain repays rollout GPU-hours and operational complexity.

I would train a pairwise scorer with domain-balanced sampling and validate calibration and ranking beyond the annotators and prompts seen in training.

• Split by prompt family, time, and annotator group to prevent paraphrase or reviewer leakage, while retaining disagreement and provenance.
• Report pairwise accuracy, calibration, domain and language slices, length bias, position bias, and adversarial reward-hacking cases.
• Before RL use, compare reward rankings with blinded human judgments on fresh policy outputs rather than only the original 500,000 pairs.

I would mix targeted long-context examples and evaluate position-sensitive tasks instead of assuming short SFT preserves 128,000-token behavior.

• Include retrieval, aggregation, and needle tasks across lengths and positions, with a controlled fraction of 16,000 to 128,000-token sequences.
• Use packing and length curriculum carefully because long batches dominate token count and memory even when examples are few.
• Gate on short-task quality, long-context retrieval, lost-in-the-middle slices, throughput, and memory before accepting the tuned model.

I would extend it only if compression and task gains outweigh embedding changes and compatibility cost.

• Train candidate tokens on domain text, require meaningful frequency and nonoverlap, and compare average length, truncation, and inference tokens against the current tokenizer.
• New embeddings need careful initialization and enough supervised or continued-pretraining exposure, while old-token behavior must remain stable.
• The new tokenizer creates a distinct model and serving artifact, so adapters, caches, quantized engines, and clients cannot silently mix versions.

The run needs about 30.3 wall-clock hours and roughly 243 H100 GPU-hours including the 20% allowance.

• Two hundred million divided by 2,200 is about 90,909 seconds, or 25.3 hours of measured compute time.
• Multiplying by 1.2 covers evaluation, checkpoints, stragglers, and expected interruptions, giving about 30.3 hours.
• Eight GPUs for 30.3 hours yield about 243 GPU-hours, which I would price with storage and failed-run reserve separately.

I would checkpoint at most every 20 minutes of progress and benchmark asynchronous save time against training throughput.

• A faithful checkpoint includes model or adapter, optimizer, scheduler, scaler, random states, dataloader position, and global step under one manifest.
• Sharded checkpoints avoid gathering a huge full state on rank zero, while periodic consolidated weights support evaluation and serving.
• Resume tests deliberately kill workers and verify matching loss and sample order, because a written checkpoint is not proof of recoverability.