LLM Engineer interview questions

An LLM repeatedly predicts a probability distribution for the next token from the tokens already in its context.

• A decoding rule selects one token from that distribution and appends it to the sequence.
• The expanded sequence is fed back into the model until a stop condition or output limit is reached.
• The model generates statistically likely continuations, not answers retrieved from a built-in factual database.

A transformer is a neural-network architecture that processes token representations through attention and feed-forward layers.

• Attention lets each token combine information from relevant tokens in the available sequence.
• Feed-forward blocks transform each token representation after attention has mixed context.
• Residual connections and normalization help many layers train and pass information reliably.

Tokenization converts text into integer token IDs from the model's fixed vocabulary.

• A token may represent a whole word, part of a word, punctuation, whitespace, or a byte sequence.
• The model consumes IDs and maps them to learned vector representations before transformer layers process them.
• Different model families can tokenize the same text differently, so token counts are tokenizer-specific.

Token counts determine how much input fits, how much output remains available, and often how an API request is priced.

• The prompt, conversation history, retrieved context, and generated answer share the model's context budget.
• Long inputs increase computation and can dilute useful evidence even when they fit technically.
• Counts should use the tokenizer of the selected model rather than a word-count estimate.

The context window is the maximum token sequence the model can consider for one generation request.

• It includes instructions, examples, user input, retrieved documents, tool results, history, and usually the generated continuation.
• Content outside the supplied window is not remembered automatically by the model.
• A larger window increases capacity but does not guarantee that every detail will be used accurately.

Attention computes how strongly one token representation should use information from other token representations in the sequence.

• Queries are compared with keys to produce weights, and those weights combine the corresponding values.
• Different attention heads can learn different relationships such as syntax, references, or local patterns.
• Attention weights are intermediate computations and should not be treated as a complete explanation of model reasoning.

Causal self-attention lets each position attend only to itself and earlier positions, not future tokens.

• A causal mask enforces this left-to-right information flow during training and generation.
• The model can train on many next-token targets in parallel even though inference emits tokens sequentially.
• Models such as Llama and Qwen use decoder-only transformer designs for autoregressive generation.

Attention alone does not encode token order, so the model needs positional information to distinguish differently ordered sequences.

• Position embeddings or rotary position encodings add order-related signals to token representations.
• This lets the model distinguish meanings where the same tokens appear in a different order.
• The positional method also influences how a model behaves near and beyond its trained context lengths.

Pretraining teaches a model broad language patterns by optimizing next-token prediction over a large text corpus.

• The model adjusts its parameters to reduce prediction loss across many examples.
• Pretraining can produce broad capabilities but does not by itself make the model follow user instructions reliably.
• Data quality, mixture, scale, and training objective all shape the resulting base model.

A parameter is a learned numeric weight that contributes to the transformations performed by the network.

• Training updates parameters to reduce the objective on training examples.
• Parameter count is one indicator of model capacity and compute cost, not a direct measure of answer quality.
• Facts are distributed across many weights rather than stored as readable records with exact source links.

A base model predicts text continuations, an instruct model is tuned to follow tasks, and a chat model is adapted to structured multi-turn roles.

• Base models often need completion-style prompting and may continue the prompt instead of answering it.
• Instruct models learn response patterns from instruction and answer examples.
• Chat models expect a model-specific template for system, user, and assistant messages.

Instruction tuning fine-tunes a pretrained model on examples that pair tasks or conversations with desired responses.

• It teaches the model to interpret requests and produce response formats more useful than raw continuation.
• The examples can cover question answering, summarization, extraction, dialogue, and refusal behavior.
• Instruction tuning improves behavior but does not guarantee factual accuracy or safety in every prompt.

Preference alignment trains a model to favor responses judged better according to human or synthetic preference data.

• Methods may learn a reward or preference signal and optimize the model toward preferred outputs.
• Alignment can improve helpfulness, style, and safety after supervised instruction tuning.
• It can also introduce biases from raters and objectives, so preferred behavior is not the same as objective truth.

A hallucination is fluent output that is unsupported, fabricated, or inconsistent with the available evidence.

• Next-token training rewards plausible continuation and does not guarantee a fact-check before every claim.
• Hallucinations can affect facts, citations, calculations, tool results, and claims about provided context.
• Grounding, retrieval, tools, constraints, and evaluation can reduce risk but cannot remove it completely.

A text embedding is a dense numeric vector designed to represent useful semantic properties of text.

• An embedding model maps a query, sentence, or document into a fixed-length vector.
• Similar meanings often produce nearby vectors according to the model's training objective.
• The vector is useful for retrieval, clustering, classification, and deduplication, but it is not human-readable text.

Semantic similarity is estimated by comparing embedding vectors with a metric such as cosine similarity or dot product.

• The query and candidate text must be encoded with the expected input format of a compatible embedding model.
• A higher score generally means closer representation under that model, not guaranteed relevance for every business task.
• Similarity thresholds and rankings need evaluation on domain examples rather than universal constants.

Stored documents and queries must be represented in the same compatible vector space for their similarity scores to be meaningful.

• Changing the embedding model usually requires re-embedding the indexed corpus.
• Some models use different query and passage prefixes or encoders while still producing a deliberately shared space.
• Vector dimensions alone do not prove compatibility because two models can have equal dimensions and unrelated geometry.

Embedding dimension is the number of numeric components in each vector and affects storage, index memory, and computation.

• Higher dimension can represent more features but does not automatically produce better retrieval.
• The dimension is fixed by the selected embedding model and must match the vector index schema.
• Model quality should be compared on retrieval evaluation, not selected by dimension alone.

A vector database stores embeddings and retrieves nearby vectors efficiently, usually together with source text and metadata.

• Approximate nearest-neighbor indexes trade a small amount of recall for faster search at scale.
• Metadata filters can restrict candidates by tenant, language, date, or document type.
• The database finds similar records; it does not itself generate an answer or guarantee relevance.

Metadata filters enforce known constraints before or during similarity search so semantically close but invalid records are excluded.

• Typical filters include access scope, tenant, locale, product, time range, and publication status.
• Filtering can improve relevance and prevent cross-tenant or unauthorized retrieval.
• Metadata needs consistent indexing and governance because missing or incorrect fields silently remove or expose candidates.