AI Research Engineer interview questions

I would turn the idea into a falsifiable claim with a fixed comparison, metric, and decision rule before seeing results.

• I would state the intervention and mechanism, such as RoPE scaling improving 32K retrieval because relative phases remain usable beyond the 8K training length.
• I would freeze the baseline, data snapshot, token budget, three seeds, primary metric, and subgroup metrics in the experiment record.
• I would set a threshold such as at least +1.0 accuracy point with a 95% confidence interval excluding zero, rather than accepting any positive delta.
• I would label later analyses as exploratory and require a fresh confirmation run before treating them as evidence.

I would run a 2 by 2 design so the main effects and the interaction are identifiable instead of testing each change only against baseline.

• The four cells are neither change, A only, B only, and A plus B, all matched on data order, tokens, initialization policy, and evaluation.
• With three seeds per cell, I would estimate A, B, and the interaction A:B rather than adding two independently measured deltas.
• A positive interaction means the combined gain exceeds the sum of main effects; a negative one can reveal that both changes solve the same bottleneck.
• If compute permits only eight runs, I would reduce secondary evaluations before dropping cells, because a missing cell destroys the factorial comparison.

I would budget from the minimum detectable effect and observed variance, not from an arbitrary number of runs.

• I would estimate per-example or per-seed variance from a pilot, set alpha to 0.05 and power to 80%, then solve for the required examples and seeds.
• Pairing both models on the same evaluation items often cuts variance, so 3,000 paired items may be more informative than 6,000 independently sampled items.
• I would reserve roughly 10% of compute for failed starts and one untouched confirmation, while keeping that reserve outside the primary analysis.
• If the required budget is impossible, I would raise the detectable-effect threshold or narrow the hypothesis instead of running an underpowered test silently.

A paired test removes item-difficulty variation by comparing both models on the same independent evaluation items.

• The pair is the item ID: I would store both outcomes for each item and analyze the within-item delta rather than two independent aggregate means.
• For binary correctness, McNemar's test uses discordant item pairs; for continuous scores, a paired permutation test or paired bootstrap preserves item covariance.
• Matching an integer decoding seed does not create the pairing, because different token distributions can map the same random stream to different draws.
• For stochastic generation, I would take repeated samples per item or use a justified common-random-number coupling, then model uncertainty across both items and samples.

I would use a two-level analysis that represents uncertainty from independent training runs and from the sampled evaluation items.

• I would build a crossed design in which each baseline and candidate seed block is scored on the same item set, preserving method pairing wherever the seed policy supports it.
• A two-level bootstrap resamples seed pairs and item IDs independently, then recomputes the overall method delta for every replicate.
• A crossed random-effects model is another option, but treating every seed-item score as an independent row would create pseudoreplication and an interval that is too narrow.
• I would report the interval with seed and item variance components, and add seeds if the run-level component is too poorly estimated for the decision.

I would define the hypothesis family before looking at results and correct within that family rather than celebrating the smallest raw p-value.

• Holm correction is a good default for a small set of confirmatory claims because it controls family-wise error and is less conservative than Bonferroni.
• Benjamini-Hochberg fits a broader screening sweep where accepting a controlled false discovery rate, such as 5%, is reasonable.
• I would nominate one primary benchmark and metric, leaving the other five as secondary evidence rather than multiplying co-primary claims.
• The report would include raw and adjusted p-values, effect sizes, and confidence intervals so correction does not replace judgment about magnitude.

I would frame the quality comparison as a preregistered non-inferiority test with a margin of 0.3 points.

• I would define delta as candidate minus baseline, verify that the candidate actually consumes 75% of the stated total FLOPs, and set the non-inferiority margin at -0.3 before running either arm.
• The null is delta less than or equal to -0.3, so I would declare non-inferiority only if the lower one-sided 95% confidence bound is above -0.3.
• Matched training-seed blocks and the same evaluation items reduce variance, while the interval must still reflect uncertainty from both seeds and items.
• I would recommend the method only if the 25% saving and latency or memory guardrails also hold, because non-inferiority proves neither equivalence nor superiority.

I would use repeated, crossed measurements to estimate which sources of randomness dominate the total variance.

• I would vary initialization seed, data-order seed, checkpoint step, and evaluation sampling separately instead of changing all four together.
• A random-effects model can attribute variance to seed, checkpoint, benchmark item, and residual terms, with interactions added when the design supports them.
• If data order explains 60% of observed variation, adding more evaluation examples will not fix uncertainty; more independent training runs will.
• I would publish variance components and the resulting seed count so later experiments spend compute on the dominant uncertainty source.

I would first prove that the intervention activated its proposed mechanism rather than interpreting an inert implementation as a scientific negative.

• Before the run, I would name a mechanistic signature such as longer attention distance, changed routing entropy, or a measurable auxiliary representation.
• Treatment and baseline traces must differ on that signature, while config assertions, gradient flow, and parameter updates confirm that the new path was actually used.
• A removal, sham, or coefficient-dose check should turn the signature off or scale it predictably, which is stronger evidence than one final metric remaining flat.
• Only after activation passes would I use the effect interval to decide whether the minimum useful gain is excluded; otherwise the result is invalid or inconclusive, not negative.

Warmup limits unstable early updates while optimizer moments and activation scales settle, and decay reduces update size as training approaches a minimum.

• I would start with linear warmup over 1% to 3% of total steps, then compare cosine and linear decay at equal peak rate, tokens, and optimizer settings.
• Too little warmup can cause loss spikes or overflow, while 10% warmup may waste a substantial fraction of a short run at an unnecessarily low rate.
• The comparison should include early loss, final validation loss, gradient norm, and quality at fixed FLOPs rather than only the final checkpoint.
• I would tune peak rate jointly with schedule shape because a cosine schedule can appear better merely because its average learning rate differs.

AdamW combines adaptive first and second moments with weight decay applied separately from the gradient update, so its exact hyperparameters are part of the result.

• The update uses bias-corrected moment estimates with beta values often near 0.9 and 0.95 or 0.999, and changing beta2 alters responsiveness to gradient variance.
• Decoupled decay shrinks parameters directly; it is not equivalent to adding L2 loss under an adaptive preconditioner.
• Biases and normalization scales are often excluded from decay, while embedding and output weights need an explicit documented policy.
• Epsilon, precision, gradient clipping, and whether moments remain in FP32 can change low-magnitude updates, so I would checkpoint and report them all.

Gradient noise scale estimates where a larger batch stops buying proportional optimization efficiency because gradient variance is already small relative to the mean signal.

• Below the critical batch, doubling batch size can often support an approximately doubled learning rate and reduce the steps needed for the same token exposure.
• Above it, extra examples mostly duplicate gradient information, so throughput may rise while quality per training FLOP falls.
• I would estimate the scale from gradient statistics across microbatches at several checkpoints because it changes during training.
• A batch study must match total tokens and retune learning rate; comparing equal steps gives the larger batch an unfair compute advantage.

Label smoothing moves some target mass away from the one-hot class, discouraging extreme logits and often improving calibration.

• With smoothing 0.1 in a K-class task, the target is mostly the correct class while the remaining mass is distributed across other classes according to the chosen convention.
• It can improve validation accuracy when labels are noisy, but too much smoothing weakens the signal for rare or fine-grained classes.
• It may hurt knowledge distillation or confidence-based selection because the model is explicitly trained not to match sharp target probabilities.
• I would sweep values such as 0, 0.05, and 0.1 and report accuracy, NLL, ECE, and per-class recall rather than accuracy alone.

Focal loss is useful when easy examples dominate training and the experiment needs to emphasize hard or rare examples.

• It multiplies cross-entropy by (1 - p_t) raised to gamma, so gamma 2 strongly downweights examples the model already classifies confidently.
• An alpha weight can address class frequency separately, but tuning alpha and gamma together can overcorrect imbalance.
• Hard examples include mislabeled data, so focal loss may amplify annotation noise and reduce probability calibration even when recall improves.
• I would compare it with class-weighted cross-entropy at matched sampling and report PR-AUC, rare-class recall, NLL, and gradient mass by class.

Temperature scales similarity logits before softmax and controls how strongly the loss concentrates on the hardest negatives.

• In InfoNCE, dividing cosine similarities by 0.07 creates sharper probabilities and larger gradients around close negatives than a temperature of 0.2.
• A very low value can make a few false negatives dominate and destabilize training, while a high value may provide gradients that are too diffuse.
• The best value depends on embedding normalization, batch size, and negative mining because all three change the similarity distribution.
• I would track retrieval Recall@k, alignment and uniformity statistics, gradient norms, and false-negative slices, while constraining a learned temperature to a safe range.

I would combine hard-label supervision with a temperature-softened teacher-to-student divergence and isolate its value at fixed student conditions.

• A standard objective is (1 - alpha) times CE plus alpha times T^2 * KL(softmax(z_teacher / T) || softmax(z_student / T)).
• The T^2 factor compensates for the gradient shrinkage caused by softening, while values such as T from 2 to 4 expose secondary class preferences.
• Alpha trades task labels against teacher behavior, and feature matching is a separate hypothesis that should not be bundled into the first comparison.
• I would compare with the same-size student trained without a teacher, reporting quality, calibration, and slices while accounting separately for teacher-logit generation.

I would choose the auxiliary coefficient from gradient behavior on shared parameters, not from the two scalar loss values alone.

• On the shared backbone I would compare ||g_main|| with ||lambda * g_aux||; parameters used only by the auxiliary head do not belong in that ratio.
• I would also measure cosine similarity between those same gradient vectors, because cosine shows direction while the norm ratio shows whether an aligned or conflicting signal is large enough to matter.
• A coefficient sweep can target a bounded auxiliary-to-main norm ratio, with clipping or a schedule added only as a separately controlled intervention.
• I would retain the term only if the main task improves at fixed total compute and a removal ablation reverses the claimed effect without hidden slice regressions.

I would predeclare whether the claim is about matched model training work or matched full algorithmic compute, because those are different comparisons.

• A matched-token view can hold data exposure and model forward-backward FLOPs fixed, but it must not be described as equal total FLOPs when optimizer work differs.
• Full compute adds recomputation plus optimizer updates, including preconditioner construction and application for methods such as Shampoo, to the forward-backward ledger.
• Every optimizer receives the same number of tuning trials for learning rate, decay, and its method-specific settings rather than inheriting AdamW defaults.
• I would plot validation loss against both cumulative forward-backward FLOPs and cumulative full FLOPs, alongside step time, peak memory, and seeded final quality.

Pre-norm places normalization before each residual branch and usually gives deep Transformers a cleaner gradient path, while post-norm normalizes after residual addition.

• In pre-norm, the identity residual path bypasses the sublayer and normalization, which helps gradients propagate through dozens or hundreds of layers.
• Post-norm can produce stronger representation changes per block but is more sensitive to initialization, warmup, and depth.
• Pre-norm is not automatically better at matched compute; its residual stream can grow, and final normalization becomes important.
• I would compare both with matched parameters and tokens, then inspect loss spikes, layerwise gradient norms, activation scale, and final benchmark quality.

RoPE rotates query and key features by position-dependent angles, while ALiBi adds head-specific linear distance penalties to attention scores.

• RoPE makes dot products depend on relative phase while retaining rich positional geometry, but unseen long positions can rotate beyond the trained frequency regime.
• ALiBi has no learned position table and biases heads toward different locality ranges, often extrapolating simply but with a less expressive positional interaction.
• I would train both to 8K with matched parameters and tokens, then evaluate perplexity, retrieval, and reasoning at 8K, 16K, and 32K.
• Any RoPE interpolation or frequency rescaling is a separate intervention, and I would verify short-context quality because improving 32K can damage the trained range.