AI Research Engineer interview questions

A good research question names the intervention, the measured outcome, and the conditions of the comparison.

• Here the intervention is adding a defined amount and type of synthetic data.
• The outcome could be macro-F1 on a fixed, untouched test set rather than a vague claim of better quality.
• The conditions should fix the model, real training data, compute budget, and evaluation protocol so the question is answerable.

I would make a directional, measurable prediction before running the comparison.

• For example, AdamW will reach validation loss below 0.40 in fewer training steps than SGD under the same data and compute budget.
• The optimizer is the changed factor, while the threshold and number of steps make the result observable.
• The claim is falsifiable because AdamW can fail to reach the threshold first or fail to reach it at all.

The null hypothesis says the new variant has no real effect on the chosen outcome relative to standard attention.

• If the outcome is validation loss, the null can state that the mean loss difference across seeds is zero.
• An observed small difference may still be compatible with random variation under this hypothesis.
• The alternative hypothesis states the expected nonzero or directional effect that the experiment seeks to support.

The claim is the conclusion the authors want readers to accept, while the evidence is the measurement supporting it.

• Two points higher accuracy is evidence only after the dataset split, metric, baseline, and evaluation protocol are specified.
• The claim might be narrow, such as improvement on one benchmark, or broader, such as better generalization across tasks.
• A strong reading checks whether the tables, uncertainty, and ablations actually support the breadth of the stated claim.

Reproduction usually re-creates a result with the original method and artifacts, while replication tests the claim with an independent setup.

• A reproduction might run the authors' code on the same dataset and compare the reported metric.
• A replication might use a fresh implementation, a new sample, or another benchmark that tests the same hypothesis.
• Terminology varies between fields, so a report should state exactly which code, data, and protocol were reused.

A baseline gives a realistic reference to beat, while an oracle estimates an upper bound using information the real method cannot access.

• A majority-class predictor or the standard published model can serve as a baseline.
• An oracle might choose the correct candidate using ground-truth labels, so it is informative but not a valid usable method.
• If a proposal barely beats the baseline and remains far below the oracle, the two references reveal both its gain and remaining headroom.

A controlled experiment changes the method of interest while holding other influential conditions fixed.

• Both runs should use the same model size, data split, training budget, and evaluation code.
• The control can be the established training recipe, and the treatment changes only the proposed method.
• Repeating both conditions across matched seeds helps separate the method's effect from training randomness.

Learning rate is the independent variable, and validation loss is the dependent variable.

• The researcher deliberately sets values such as 0.001, 0.0003, and 0.0001 for the independent variable.
• Validation loss is observed after training and may change in response to those settings.
• Model architecture, split, batch size, and step count should remain controlled rather than becoming extra independent variables.

A confound is an uncontrolled factor that changes with the treatment and could explain the measured difference.

• If model A trains for 10,000 steps and model B for 20,000, training budget is confounded with architecture.
• Better accuracy from model B cannot then be attributed only to its architecture.
• Matching the step or compute budget removes that alternative explanation from the comparison.

Multiple seeds show whether a result persists across ordinary randomness in training.

• A seed can affect parameter initialization, data order, dropout masks, and sampled outputs.
• Reporting only the best of five seeds gives an optimistic and biased view of the method.
• Reporting the mean and spread across the same seed set for both methods makes the comparison more credible.

The spread shows that the measured accuracy depends partly on stochastic training choices, not only on the recipe.

• The mean is 80%, but individual runs differ by up to two percentage points from it.
• A new method scoring 81% once is not convincing against this observed run-to-run variation.
• More repeated runs provide a better estimate of the typical result and its uncertainty.

A 95% confidence interval summarizes the estimate's precision under the sampling assumptions of the calculation.

• A narrow interval means repeated measurements estimate the mean more precisely than a wide interval.
• Strongly overlapping intervals can warn that an apparent mean difference is uncertain, though overlap alone is not a formal test.
• In the frequentist interpretation, 95% refers to the long-run coverage of the procedure, not a 95% probability that this fixed interval contains the mean.

Effect size states how large the difference is, while statistical significance addresses how compatible the data are with a null model.

• With enough samples, a 0.05-point accuracy gain can be statistically significant but scientifically unimportant.
• An absolute metric difference or a standardized measure such as Cohen's d makes the magnitude visible.
• The useful judgment combines magnitude, uncertainty, and the cost or relevance of obtaining the gain.

Running many comparisons raises the chance of finding an apparently significant result by luck.

• At a 0.05 threshold, each null comparison has a 5% false-positive risk under its assumptions.
• Selecting the smallest p-value after seeing all 20 results hides the size of that search.
• Pre-registering the main comparison or applying a correction such as Bonferroni makes the evidence more honest.

A packed causal-LM loss mask should count real next-token targets while excluding padding and artificial transitions between independent documents.

• After the causal shift, valid target tokens contribute to loss and ignored label positions use a value such as -100.
• If document B directly follows document A, keep an intended EOS target but mask B's first-token target unless the A-to-B transition belongs to the objective.
• The loss mask does not stop B from attending to A, so independent documents also need a segment-aware or block-diagonal causal attention mask.

Per-sequence normalization gives each sequence equal weight, while per-nonpadding-token normalization gives each valid target token equal weight.

• With token normalization, a 400-target sequence contributes four times as much as a 100-target sequence before averaging.
• With sequence normalization, each sequence's valid-token mean is computed first, so short and long sequences affect the batch equally.
• Padding and ignored labels must be absent from both numerator and denominator, and the chosen rule must stay fixed across compared runs.

KL divergence measures the extra log-probability cost of using one distribution to represent samples from another.

• KL(P || Q) weights log(P divided by Q) by probabilities from P.
• It is zero when the distributions match but is asymmetric, so KL(P || Q) usually differs from KL(Q || P).
• In distillation, it can train a student distribution to match the teacher's probabilities rather than only hard labels.

Logits are unrestricted raw scores, and softmax converts them into nonnegative probabilities that sum to one.

• Logits of 2, 1, and 0 preserve class preference but are not probabilities themselves.
• Softmax exponentiates relative scores and normalizes them, so adding the same constant to every logit changes no probability.
• Cross-entropy implementations usually accept logits directly because combining the operations is numerically more stable.

A gradient of None means autograd found no differentiable path from the total loss to that parameter.

• I would verify that backward receives total_loss = main_loss + aux_weight * aux_loss, because merely logging aux_loss does not connect it to the graph.
• I would inspect parameter.requires_grad, detach, item, NumPy conversion, no_grad, and conditionals that skip the auxiliary head, then confirm that aux_loss.requires_grad is true.
• torch.autograd.grad with allow_unused=True on a tiny batch can locate unused parameters; optimizer membership affects updates but not whether backward creates gradients.

The gradient tree must have the same paths and compatible leaves as the parameter tree, while parameter-shaped optimizer state must track those same leaves.

• Dict keys, tuple or list nesting, and leaves define a PyTree treedef, so adding or renaming a parameter changes the structure expected by updates.
• The optimizer state's outer tree may differ, but momentum or variance subtrees must still align with the parameter paths and shapes they track.
• I would compare tree structures and path-shape listings, then rebuild optimizer state and masks from the exact parameter tree after an architecture change.