AI Safety Engineer interview questions

I would stratify preference pairs by harm class, ambiguity, and legitimate use rather than sampling only obvious violations.

• Each harmful request gets matched with nearby benign, transformative, and boundary cases so the preferred response can distinguish refusal from safe assistance.
• I would oversample rare severe classes, then use importance weights so the training distribution does not pretend those cases dominate real traffic.
• A held-out slice by source and annotator measures safe-completion rate, refusal recall, and benign false-positive rate before the data enters training.
• I would cap repeated templates and cluster semantic duplicates to stop one jailbreak family from dominating the gradient.

I would make the policy decision-oriented, with labeled examples, escalation rules, and an explicit way to record ambiguity.

• Each harm class gets inclusion, exclusion, and borderline examples plus the expected response mode: comply, safe-complete, refuse, or escalate.
• Annotators label policy outcome and confidence separately, so a low-confidence majority is not mistaken for a clear gold label.
• I would double-label a calibrated sample, track class-level Krippendorff's alpha, and send disagreements to adjudication rather than forcing consensus in chat.
• Adjudication decisions become versioned examples in the next policy release, preserving the original labels for audit and reanalysis.

I would evaluate the reward model as a safety classifier and ranker on held-out, adversarial, and distribution-shifted preference pairs.

• Pairwise accuracy is broken down by harm class, severity, language, and comply-versus-refuse boundary instead of reported as one aggregate.
• I would test swaps, paraphrases, response-length controls, and superficial style changes to expose position, verbosity, and refusal-phrase shortcuts.
• Calibration curves compare reward margins with human agreement, since a large score gap should correspond to a genuinely unambiguous preference.
• The release check also measures downstream policy behavior because a small ranking bias can be amplified during optimization.

I would treat KL control as a safety constraint that limits drift from a known policy, not as a tuning constant chosen only for reward.

• Too little KL pressure lets the policy exploit reward-model gaps and lose pretrained uncertainty signals; too much preserves unsafe or unhelpful reference behavior.
• I would use a frozen, versioned reference close to the starting policy and plot safety metrics against reward and measured KL across checkpoints.
• Per-class evals must include over-refusal and dangerous compliance because the same average KL can hide concentrated movement in one harm class.
• I would select a checkpoint on the Pareto frontier and reject any point that crosses a hard severe-harm gate even if preference reward improves.

I would search for behaviors that raise the optimization reward without satisfying the underlying safety rubric.

• Compare reward-model scores with blinded human labels on high-reward samples, especially repetitive refusals, disclaimers, verbosity, and policy-keyword mimicry.
• Train probe policies against frozen reward snapshots and inspect the highest reward disagreements rather than random outputs.
• Use independent process checks, rule-based facts, and a second judge where possible so the detector does not share the same shortcut as the reward model.
• Track reward, severe-harm rate, safe-completion rate, and response diversity together; reward rising while external safety metrics flatten is a stop signal.

PPO instability can move a previously safe policy abruptly toward reward exploits or broad refusal even when average reward looks healthy.

• I would monitor policy KL, clip fraction, entropy, value loss, and per-class safety metrics at every checkpoint rather than only at the final epoch.
• Conservative learning rates, weaker updates, reward normalization, gradient clipping, and an adaptive KL target reduce destructive policy jumps.
• Fixed canary prompts for severe harm and benign dual-use requests run during training and halt the job when a hard boundary regresses.
• After rolling weights back to a safe checkpoint, I would reset or deliberately adjust optimizer moments and lower the learning rate or update strength; restoring the momentum that caused the unstable step can reproduce it.

DPO simplifies optimization, but its safety quality remains bounded by preference-pair coverage and the chosen reference policy.

• It avoids a separately exploitable reward model and unstable online PPO loop, which reduces one class of reward hacking.
• Label noise and one-sided refusal pairs can directly teach blanket refusal because DPO has no independent reward model to inspect.
• The beta setting controls departure from the reference, so I would sweep it against severe-harm recall, safe-completion rate, and style artifacts.
• I would compare DPO and RLHF on the same held-out adversarial set and choose by the safety Pareto frontier, not training simplicity alone.

I would validate the constitution, judge, and trained policy separately because errors in any one can become the alignment target.

• Constitution clauses need concrete conflict rules and examples; otherwise two reasonable principles can rank the same response differently.
• The AI judge is calibrated against blinded human adjudication by harm class, language, and ambiguity, with position and self-preference tests.
• I would vary judge model and constitution wording on a sample to measure how dependent labels are on one evaluator configuration.
• Final policy evals use independent judges and humans, since scoring the result with the same judge that generated preferences creates circular evidence.

I would preserve the original response, selected principle, structured critique, revision, and provenance as separate artifacts.

• A principle selector first chooses applicable clauses and records conflicts instead of sending the entire constitution into every example.
• The critic must cite the violated span and principle ID, while the reviser receives only the critique, source response, and allowed principles.
• Validators check that the revision removes the violation without inventing facts, dropping useful content, or turning every boundary case into refusal.
• Human review focuses on high-severity, low-confidence, and critic-reviser disagreement cases, which gives better coverage than uniform review.

I would use process supervision when unsafe intermediate actions matter even if the final answer appears harmless.

• Tool-use planning, bio protocols, and cyber workflows need step labels because an agent can take an irreversible unsafe action before a final-response judge sees it.
• Outcome supervision is cheaper and appropriate when success and harm are reliably observable at the end, such as a bounded classifier decision.
• Process labels cost more and can encode a preferred reasoning style, so I would supervise verifiable actions and state transitions rather than private reasoning prose.
• The eval compares both final harm and intermediate policy violations to confirm that process supervision adds signal instead of merely changing presentation.

I would decompose the task into checkable claims and actions, then combine targeted tools, model assistance, and sampled expert review.

• Evidence retrieval, executable tests, and provenance checks handle verifiable subclaims before any model judge scores the remaining judgment calls.
• Multiple independent critics inspect different failure modes, such as factuality, policy compliance, and tool authorization, instead of debating one vague score.
• Experts adjudicate high-impact disagreements and a random low-risk sample to estimate errors that confidence routing misses.
• I would benchmark the oversight method on tasks with known answers before trusting it on tasks beyond direct human capability.

Weak supervision expands coverage, but correlated labeling errors can become a confident false consensus.

• Heuristics, policy keywords, model judges, and source metadata should be modeled as separate noisy labeling functions with measured overlap and conflict.
• A small expert-labeled gold set estimates each source's class-specific accuracy rather than assigning equal votes.
• Shared model ancestry and copied rules create correlated errors, so adding three similar judges does not provide three independent signals.
• I would reserve severe and novel harm cases for expert review and report where weak labels are abstentions rather than forcing complete coverage.

I would create paired conversations where only the user's stated belief or preference changes while the underlying answer stays fixed.

• Include false-premise questions, political and personal opinions, and cases where the user pressures the model after receiving a correct answer.
• Score factual consistency, appropriate uncertainty, respectful disagreement, and whether confidence shifts without new evidence.
• Counterbalance answer order and user identity cues so the eval does not confuse politeness or demographic style with sycophancy.
• Multi-turn variants test whether repeated pressure causes gradual capitulation that a one-turn benchmark would miss.

I would train on attack families and behavioral invariants, not a list of memorized suffixes.

• The mix includes human red-team cases, PAIR-style semantic attacks, encoding transformations, role-play, multilingual variants, and benign near-neighbors.
• Split by attack generator, template cluster, and time so the holdout contains unseen strategies rather than paraphrases of training examples.
• Refresh hard examples against current checkpoints because successful attacks become stale as the model changes.
• Measure unseen-family ASR and benign safe-completion rate together to catch both brittle defense and broad refusal.

I would mix targeted safety examples with representative capability and benign dual-use data, then gate every checkpoint on both safety and utility.

• Safety data covers severe harms and boundary cases, while replay data preserves instruction following, domain competence, and safe assistance.
• Low learning rates, limited epochs, and PEFT such as LoRA can constrain parameter drift, but they do not replace capability evaluation.
• I would compare class-level ASR, over-refusal, core task scores, and calibration against the starting model after each epoch.
• If one capability regresses, I would adjust the mixture or stop at an earlier checkpoint rather than patching the loss with duplicated examples.

I would mine benign requests that resemble harmful content closely enough to trigger the current classifier, then verify them before retraining.

• Candidate sources include quoted analysis, prevention, fiction, medical triage, policy discussion, and transformed content from live false positives.
• Rank candidates near the decision boundary or by ensemble disagreement, then deduplicate them by semantic cluster before annotation.
• Annotators label both harm class and allowed intent so the model learns the distinction rather than merely down-weighting keywords.
• Keep a frozen hard-negative holdout and monitor severe-harm recall because aggressive false-positive reduction can open real bypasses.

I would map product policy to Llama Guard 2 labels explicitly, add product-specific boundary cases, and evaluate its standard discrete decision rule after fine-tuning.

• Training examples balance violating, allowed, and ambiguous conversations across user and assistant messages, with provenance and policy-version fields.
• Multi-turn and multilingual examples reflect the deployed surface, while near-duplicate templates remain in only one data split.
• Evaluation compares emitted safe or unsafe decisions and category labels with the original model, reporting per-category confusion on benign dual-use and rare severe harms.
• The standard contract does not expose a calibrated continuous score. Only if the integration deliberately exposes token logits and explicitly defines a continuous score would I apply temperature scaling or class thresholds on a held-out calibration set, then pin and validate that wrapper.

I would separate broad harm families from actionable leaf labels and allow one example to carry multiple labels.

• Parent nodes such as violence or cyber support stable reporting, while leaves such as credential theft or malware deployment drive distinct controls.
• Labels encode content, intent, and target only where each dimension changes enforcement; mixing all three into one flat label creates sparse combinations.
• Annotation guidance covers co-occurrence, primary versus secondary harm, and when a parent-only label is valid because evidence is insufficient for a leaf.
• Evaluation includes hierarchical precision-recall so a parent-correct, leaf-wrong prediction is not treated like a completely unrelated class.

I would treat taxonomy changes as schema migrations with stable identifiers, mappings, and an explicit compatibility window.

• Renames retain the same immutable ID, while splits and merges publish many-to-many mappings plus examples that define the new boundary.
• Classifiers emit taxonomy and model versions together, and dashboards can translate old labels only where the mapping is semantically valid.
• During dual-run, old and new taxonomies score the same traffic so gate owners can quantify metric discontinuities.
• Historical data remains stored under its original labels; backfills create a new derived dataset rather than rewriting audit evidence.

I would report class-level rates first and use severity weighting only as a transparent secondary decision metric.

• Each harm class receives documented impact and exposure weights, with ranges or sensitivity analysis where values are uncertain.
• Expected risk can combine violation probability, severity, and deployment exposure, but catastrophic classes still keep hard maximum thresholds.
• A weighted average must not let many low-severity improvements cancel one severe regression, so the scorecard shows both components and worst-class results.
• Weights are versioned and approved with policy owners because changing them can alter a release decision without any model change.