NLP Engineer interview questions

The scaling keeps attention logits in a range where softmax still has useful gradients as the key dimension grows.

• Unscaled dot products have variance proportional to the key dimension, so large dimensions produce extreme logits.
• Extreme logits make softmax nearly one-hot, causing most derivatives to approach zero.
• Dividing by the square root of the key dimension normalizes the variance without changing the learned ranking of keys.

Multiple heads let the model represent different token relationships in parallel rather than forcing one attention map to carry every pattern.

• One head may emphasize local syntactic links such as adjective-to-noun agreement while another follows long-range coreference.
• Separate learned query, key, and value projections give each head its own lower-dimensional representation subspace.
• Head roles are not guaranteed or perfectly interpretable, and redundant heads can often be pruned with little quality loss.

I would benchmark task-appropriate checkpoints because RoBERTa and DeBERTa often improve quality over BERT but can cost more memory or latency.

• BERT is a stable baseline with broad checkpoint and tooling support, especially when compatibility matters.
• RoBERTa changed pretraining through more data, dynamic masking, and removal of next-sentence prediction, usually yielding stronger representations at comparable size.
• DeBERTa separates content and position attention and often scores higher, but its implementation and serving profile must fit the production stack.
• I compare validation quality, p95 latency, memory, tokenizer coverage, and domain-specific checkpoint availability rather than choosing from benchmark rank alone.

An encoder-decoder model is the natural choice when the output is a new variable-length sequence, while an encoder-only model is simpler for labels, spans, or embeddings.

• Translation and abstractive summarization require autoregressive decoding conditioned on an encoded source, which models such as T5 or BART provide.
• Classification and token labeling usually need only an encoder plus a small task head, avoiding decoder latency and exposure to generation errors.
• Extractive summarization can also remain encoder-only because it selects source sentences instead of generating text.

A linear-chain CRF scores the whole label sequence and learns transition scores between adjacent tags instead of predicting each token independently.

• An ordinary CRF with Viterbi does not guarantee valid BIO output; I add a transition mask or set illegal transition scores to negative infinity before decoding.
• The gain is most useful when label transitions are strong and training data is limited; large encoders may already learn much of this structure.
• A CRF adds training and decoding complexity and does not solve nested or discontinuous entities because it still emits one linear tag per token.

Token classification assigns BIO-style labels to tokens, while a span model scores candidate start-end ranges and their entity types.

• Token labeling is computationally efficient and works well for flat entities, but subword alignment and invalid tag transitions need care.
• Span scoring represents entity boundaries directly and can classify overlapping candidates, which makes nested NER more natural.
• Enumerating all spans is quadratic in sequence length, so practical systems cap span width or prune candidates.
• I choose using the annotation structure and latency budget, then evaluate exact entity spans rather than token accuracy.

A single BIO label per token cannot represent overlapping entity layers or one entity split across nonadjacent text segments.

• Nested entities, such as New York inside New York University, need span classification, layered taggers, or hypergraph-style decoding.
• Discontinuous mentions, common in clinical text, need models that link multiple spans or predict mention fragments and their relations.
• The annotation schema must define whether inner entities and split mentions are required, because architecture and scoring follow that decision.

The encoder produces a score for every token as a possible start and another as a possible end, then selects a valid high-scoring pair.

• Training commonly applies cross-entropy to the gold start and end positions using contextual token representations.
• Inference considers pairs where end is not before start and often enforces a maximum answer length.
• Independent start and end heads are efficient but can favor incompatible boundaries, so joint span scoring or reranking can improve difficult cases.

The model needs an explicit no-answer score that is compared with the best candidate span using a tuned threshold.

• BERT-style systems often use the classification token as the null span and compare its score with the best non-null start-end score.
• I tune the score difference threshold on a validation set to balance unanswered-question recall against false answers.
• Evaluation must include answerable and unanswerable slices because aggregate exact match can hide a model that refuses too often.
• Calibration and passage retrieval coverage should be checked separately before changing the threshold.

I would preserve the taxonomy structure in prediction and loss so that a child label cannot be emitted without a compatible parent.

• A top-down cascade narrows candidates at each level and is interpretable, but an early parent error blocks every descendant.
• A single model with hierarchical constraints can share evidence across levels and enforce valid paths during decoding.
• I report metrics at each depth plus path accuracy because a correct parent and wrong leaf is different from a completely wrong branch.

Multilabel classification predicts each label independently or with modeled dependencies because several labels may be correct for one document.

• The output normally uses one sigmoid per label with binary cross-entropy, not a softmax that forces probabilities to sum to one.
• Thresholds should be tuned globally or per label because rare labels often need a different operating point.
• Micro F1 emphasizes common labels, while macro F1 exposes weak performance on rare labels.
• If labels co-occur systematically, classifier chains or a shared representation with dependency modeling can capture that signal.

I would split the document into meaningful chunks, encode them, and aggregate chunk evidence while preserving enough document structure for the label.

• For sparse evidence, max pooling or top-k attention over chunk logits can surface one decisive paragraph.
• For distributed evidence, a second-level transformer or recurrent aggregator can model the ordered chunk representations.
• Training should sample complete documents, not treat chunks as independently labeled examples when the label only applies globally.
• I validate chunk size, overlap, and truncation on long-document slices because random short examples hide coverage failures.

I would use a sparse-attention encoder when decisions depend on interactions across distant sections that chunk aggregation loses.

• Sliding-window attention reduces quadratic cost for local context, while selected global tokens connect document-wide information.
• Longer input still raises memory and latency, and pretrained checkpoints may be weaker or less domain-specific than standard encoders.
• Chunking remains simpler for independently informative sections and supports selective processing of only relevant parts.
• I compare quality on cross-section examples and p95 serving cost rather than assuming a larger context is automatically better.

I use a pooling strategy that was aligned with the model's training objective rather than assuming the first token is always a semantic sentence vector.

• Mean pooling over non-padding tokens is a strong baseline for sentence-transformer models and uses information from the whole sequence.
• First-token pooling can work when the checkpoint was explicitly trained around that representation, but vanilla masked-language pretraining does not guarantee it.
• I usually L2-normalize vectors for cosine search and verify performance on domain pairs, including short and long sentence slices.

I use a bi-encoder to retrieve or compare many candidates cheaply and a cross-encoder when a smaller candidate set needs more accurate pairwise scoring.

• A bi-encoder computes reusable vectors independently, so millions of documents or deduplication candidates can be indexed and searched efficiently.
• A cross-encoder jointly attends to both texts and captures fine token interactions, but it must run once per pair.
• A practical search stack retrieves top candidates with a bi-encoder and reranks perhaps 20 to 100 with a cross-encoder.
• For offline deduplication, blocking rules or approximate search first prevent quadratic all-pairs cross-encoding.

BM25 rewards lexical term overlap, while dense retrieval matches learned semantic representations, so each fails on different query types.

• BM25 is strong for exact product codes, rare names, and newly indexed terms without model retraining.
• Dense retrieval handles paraphrases such as waterproof hiking shoes versus rain-resistant trail footwear, but can miss exact identifiers.
• BM25 scores depend on term frequency, inverse document frequency, and length normalization; dense quality depends on training pairs and domain coverage.
• I evaluate them by query slice and often fuse ranked results when both semantic and exact-match intent matter.

Contrastive training pulls matched text pairs together and pushes mismatched pairs apart in embedding space using a similarity-based objective.

• With a multiple-negatives ranking loss, the other targets in the batch act as negatives for each query, providing many comparisons without extra encoding.
• Larger batches usually strengthen the signal, but false negatives arise when another batch item is actually relevant or equivalent.
• Temperature controls how sharply the loss emphasizes close competitors and must be tuned with embedding normalization.
• Batch construction should avoid grouping duplicate positives as negatives and should reflect the production matching task.

A useful hard negative looks plausible to the current retriever but is truly irrelevant under the task's labeling policy.

• BM25 or an earlier dense model can mine top-ranked nonpositive results that share vocabulary or topic with the query.
• Random negatives are often too easy and teach little once the model separates unrelated documents.
• Unverified top results can be false negatives, so I filter known positives and manually audit or cross-encode a sample.
• I refresh mined negatives during training because examples that were hard for the initial model may stop being informative.

Domain-adaptive pretraining continues the encoder's original self-supervised objective on unlabeled in-domain text before supervised fine-tuning.

• Masked-language modeling on clinical notes or legal filings teaches domain terminology and usage without requiring task labels.
• It is most useful when the target corpus differs substantially from general pretraining data and labeled examples are limited.
• I hold out downstream validation data, mix some general text if forgetting appears, and compare against simply adding more labeled examples.
• The tokenizer remains a constraint, so severe fragmentation of domain terms may justify vocabulary adaptation with careful embedding initialization.

Layer-wise decay gives lower encoder layers smaller updates and upper layers plus the task head progressively larger ones.

• For example, I might start the head at 2e-5 and multiply the rate by 0.8 for each layer toward the embeddings, so general lexical features move less than task-specific features.
• It often helps with a small labeled dataset or a task close to pretraining, where large updates to lower layers can erase useful representations.
• It can hurt under a strong domain shift if the lower layers also need to adapt, so I compare it with one uniform learning rate rather than assuming decay always wins.
• I tune the head rate and decay factor together, then choose them on held-out task quality and training stability across more than one seed.