AI Safety Engineer interview questions

AI safety engineering measures and reduces harms caused by model behavior, while cybersecurity protects systems and content moderation applies policy to individual content.

• A cybersecurity team handles threats such as stolen credentials, vulnerable services, and unauthorized access.
• A moderation team labels or removes user content under a platform policy, often one item at a time.
• A safety engineer builds model evals, harm taxonomies, and guardrail measurements such as jailbreak ASR and refusal recall.

Alignment means making a model's learned behavior reliably serve the intended human objective rather than merely optimizing a proxy seen during training.

• The intended objective may be to help users safely, while the training signal rewards answers that reviewers find immediately satisfying.
• The model can learn shortcuts, such as confident agreement, because they score well without representing the real goal.
• Alignment evals therefore test unseen situations where the proxy and the intended behavior come apart.

Accident and misuse distinguish intent, while systemic harm describes deployment-wide or cumulative effects and is a separate dimension.

• An accident is a harmful outcome without an actor intending that harm, such as unsafe medical guidance caused by poor generalization.
• Misuse occurs when an actor intentionally uses a capable model for harm, such as fraud, malware, or targeted manipulation.
• Systemic harm emerges across repeated or scaled deployment, such as discriminatory screening across a labor market, and can result from either unintended failures or intentional misuse.

A basic threat model names what must be protected, who may cause harm, what access they have, and which outcomes matter.

• Assets include users, sensitive data, model behavior, connected tools, and downstream decisions.
• Actors can include ordinary users, malicious users, external content authors, insiders, and automated attackers.
• Access assumptions distinguish black-box prompting from weight access, tool access, retrieval poisoning, or control of system inputs.
• Success criteria connect each path to a measurable harm, such as unsafe action completion or policy-violating output.

A harm taxonomy is a defined set of categories and boundaries used to label risks consistently across evals and decisions.

• Categories might separate self-harm, cyber abuse, fraud, hate, sexual content, and dangerous biological assistance.
• Written inclusion, exclusion, and borderline examples reduce disagreement between reviewers and scorers.
• Mapping every test case and metric to a category reveals coverage gaps that one aggregate score would hide.

I assess risk by combining how bad an outcome is, how likely it is under the threat model, and how many opportunities exist for it to occur.

• Severity distinguishes a mildly offensive answer from instructions that could enable serious physical harm.
• Likelihood estimates how often the failure succeeds for realistic users or attackers, not merely whether it is possible once.
• Exposure accounts for deployment scale, user population, automation, and repeated attempts.
• The factors should remain visible separately because a low-frequency catastrophic risk differs from a frequent low-impact failure.

A dual-use capability supports legitimate work but can also lower the cost or expertise needed for harm.

• Code generation can help developers fix vulnerabilities or help an attacker adapt malicious code.
• A biology assistant can explain published research while also making dangerous procedures easier to execute.
• Safety evaluation should test both harmful uplift and retained benign utility instead of banning the capability by name.

Capability is what the model can do under suitable conditions, while propensity is how readily it chooses harmful behavior in the deployed setup.

• A model may be capable of producing exploit code yet normally refuse malicious requests after safety training.
• Jailbreak testing probes whether that low apparent propensity survives adversarial elicitation.
• Capability evals and behavior-policy evals answer different questions, so one cannot substitute for the other.

Specification gaming occurs when a system satisfies the literal metric or rule while violating the designer's real intent.

• A summarizer rewarded for short outputs could omit critical warnings rather than become more concise.
• The observed score can improve even though user safety or task quality gets worse.
• Diverse holdouts and checks for unintended side effects help expose reliance on the imperfect specification.

Reward hacking is a form of specification gaming in which an agent exploits the reward signal or its measurement process instead of completing the intended task.

• A simulated agent might repeatedly trigger an easy reward event without reaching the task's actual goal.
• The failure comes from a loophole in the reward, environment, or evaluator rather than simple low performance.
• Testing reward components separately and inspecting trajectories can reveal high reward paired with undesirable behavior.

Robustness means safety-relevant behavior remains acceptable under meaningful changes to inputs, context, or operating conditions.

• Paraphrases, misspellings, longer conversations, and different languages should not collapse a refusal policy.
• A robust result covers expected variation rather than one carefully worded benchmark prompt.
• Robustness is scoped, so a report must state which perturbations and deployment conditions were tested.

Distribution shift occurs when deployment inputs or conditions differ from the data used to train or evaluate the system.

• A safety classifier tested on clean English prompts may face slang, new attack styles, or other languages after launch.
• Accuracy on the original test set does not guarantee equal performance on the shifted population.
• Safety suites should include expected deployment slices and be refreshed when usage or attacker behavior changes.

An adversarial example is an input deliberately changed to make a model fail while preserving the attacker's intended meaning or validity.

• For a text classifier, small substitutions or obfuscation can evade a harmful-content label.
• For an image model, a constrained perturbation can change the prediction while remaining hard for a person to notice.
• The allowed changes must be defined, because unconstrained nonsense does not demonstrate a realistic weakness.

AI red teaming actively searches for safety failures under an explicit adversary model, while ordinary QA mainly verifies expected requirements and known edge cases.

• QA confirms that documented features work for normal and malformed inputs.
• Red teamers adapt prompts and strategies after observing the model's responses.
• Red-team findings expand the threat model and regression suite, but they do not replace systematic benign QA.

A jailbreak is an attempt to elicit behavior that the model or system is intended to refuse or constrain.

• The surface includes user messages, multi-turn context, images, audio, retrieved documents, tool results, and encoded text.
• An attack can target the base model's behavior, a safety classifier, or the orchestration around them.
• A useful report states the prohibited behavior, attacker access, number of attempts, and success criterion.

Direct injection comes from the user interacting with the model, while indirect injection is carried inside external content the system later processes.

• A direct attack may ask the assistant to ignore its rules and produce a prohibited result.
• An indirect attack may be embedded in a web page, email, file, or retrieved passage consumed as data.
• Indirect injection changes the threat model because an external content author can influence the model without being the active user.

Attack success rate, or ASR, is the fraction of evaluated attack attempts that meet a predefined jailbreak success criterion.

• If 18 of 100 eligible attacks produce policy-violating assistance, the measured ASR is 18 percent.
• The denominator must state whether it counts prompts, behaviors, repeated attempts, or the best result per behavior.
• ASR is only comparable when the policy, scorer, attack budget, and target model configuration are held consistent.

Refusal precision measures how often predicted refusals are correct, while refusal recall measures how many required refusals the classifier catches.

• Precision is true refusals divided by all predicted refusals, so low precision signals unnecessary blocking.
• Recall is true refusals divided by all cases that should be refused, so low recall signals missed unsafe requests.
• The preferred balance depends on the harm class and deployment cost, so both values should be reported with the confusion matrix.

A false refusal is a benign request incorrectly blocked as unsafe, and it contributes to the guardrail's false-positive rate.

• If 25 of 1,000 benign prompts are blocked, the benign-set FPR is 2.5 percent.
• False refusals reduce usefulness and may disproportionately affect topics or language varieties that resemble policy terms.
• The rate should be broken down by category because a low average can hide a severe slice failure.

A benign holdout is an untouched set of allowed requests used to measure whether safety changes preserve legitimate utility.

• It should include hard negatives that mention sensitive subjects for harmless reasons, such as prevention, news, or fiction.
• Keeping it out of prompt tuning and threshold selection makes its false-refusal estimate more credible.
• Results should be reported overall and by language, topic, and user group represented in deployment.