MLOps Engineer interview questions

A pipeline orchestrator separates workflow planning and state management from task execution and artifact storage.

• Its parser or compiler turns a pipeline definition into a graph and validates structural constraints.
• Its scheduler finds runnable task instances from dependencies, trigger rules, quotas, and run state.
• Its executor submits work to local processes, worker queues, or a cluster without performing the task itself.
• Its metadata store records runs and task states, while an object store or database holds large outputs.

A DAG defines a finite partial order in which directed edges express task dependencies without cycles.

• A task becomes eligible only after its upstream dependencies satisfy their configured trigger rules.
• Independent branches may run concurrently because the graph does not impose an order between them.
• A fan-out creates parallel branches, while a join waits for the required upstream branches.
• A cycle is invalid because no topological execution order can satisfy circular dependencies.

Compilation builds a validated executable specification, while runtime creates concrete runs and resolves their state.

• Compilation captures components, edges, parameter placeholders, conditions, and execution settings in an intermediate representation.
• Static checks can reject missing inputs, incompatible types, cycles, or malformed component definitions before submission.
• Runtime binds actual parameter values, creates task instances, and evaluates conditions whose values come from earlier tasks.
• The runtime records state transitions and delegates eligible work to the configured execution backend.

A task boundary should enclose one independently executable step with an explicit, stable input and output contract.

• Separate steps that need different images, resource profiles, retry policies, owners, or cache lifecycles.
• Keep tightly coupled in-memory operations together when materializing an intermediate would add cost without reuse.
• Make side effects such as publishing a model distinct from pure transformations so their execution policy is visible.
• Avoid tiny tasks whose scheduling and container startup overhead exceeds their useful work.

Tasks should pass durable artifact references through the orchestrator and store large payloads outside its metadata database.

• A producing task writes an immutable object or versioned dataset to shared storage before publishing its URI.
• The output record should include type, schema or format, checksum, producer run, and relevant lineage metadata.
• A consuming task reads only its declared inputs, which keeps dependencies explicit and execution environments isolated.
• Atomic publication through a final rename, manifest, or commit marker prevents consumers from observing partial output.

A schedule creates runs for logical time intervals, while an event creates runs from the arrival of a defined external fact.

• Scheduled runs should use the logical interval as an input instead of assuming wall-clock execution time.
• Catchup and backfill materialize missed historical intervals according to the same schedule semantics.
• Event triggers should carry a stable event identifier and payload reference so duplicate delivery can be deduplicated.
• Both trigger types should map to the same parameterized pipeline contract so execution remains reproducible.

A pipeline task is idempotent when repeated execution for the same logical inputs converges on the same committed result.

• Derive output locations or operation keys from the run partition, input versions, and relevant parameters.
• Use upserts, compare-and-set operations, or uniqueness constraints for external writes instead of unconditional appends.
• Write temporary output first and expose it only through an atomic commit step.
• Record completed side effects under an idempotency key so a retry can recognize prior completion.

Retries should be limited to transient failure classes and use bounded exponential backoff with jitter.

• Retry policies need an attempt limit so permanent failures do not consume capacity indefinitely.
• Exponential delays reduce pressure on a recovering dependency, while jitter prevents synchronized retry waves.
• Validation errors, incompatible schemas, and invalid parameters should fail without retry because repetition cannot change them.
• Retry safety depends on task idempotency and on each attempt using the same logical input contract.

Pipeline orchestration needs separate time limits for waiting, task execution, and the overall workflow run.

• A queue timeout bounds how long a task may wait for a worker or scarce resource before it is marked unsuccessful.
• An execution timeout bounds the task process after it actually starts and should allow graceful termination before forced termination.
• A sensor or external-operation timeout limits total elapsed waiting across repeated checks or deferrals.
• A workflow timeout caps the end-to-end run even when every individual task remains within its own limit.

A task cache reuses outputs only when a cache key represents every input that can affect the result.

• The key normally includes component code or image digest, input artifact versions, parameters, and relevant configuration.
• Cached artifacts must remain immutable and accessible for at least as long as their metadata entries.
• Nondeterministic inputs such as current time, random seeds, or mutable external tables must be pinned or included in the key.
• Caching should be disabled for intentional side effects such as notifications or model publication.

Pipeline resume skips already committed tasks, while task checkpointing restores progress inside one long-running computation.

• The orchestrator resumes from persisted task states and verifies that successful outputs are still available.
• A resumed run should preserve the original parameters and input versions unless it is deliberately created as a new run.
• Training code writes periodic checkpoints containing model, optimizer, scheduler, and progress state to durable storage.
• A task restored from a checkpoint must still publish its final artifact through the normal atomic output contract.

The Airflow scheduler turns parsed DAG definitions and timetable intervals into runnable task instances.

• DAG processors parse files and serialize DAG structure so scheduling does not depend on importing code in workers.
• The scheduler creates DagRuns, evaluates task dependencies and trigger rules, and moves eligible instances toward the queued state.
• It enforces limits such as pools, DAG concurrency, and maximum active runs before handing tasks to the executor.
• Multiple schedulers coordinate through the metadata database so scheduling decisions remain consistent.

An Airflow executor maps queued task instances onto execution capacity, while workers run the task processes.

• LocalExecutor launches task processes on the scheduler host and fits a single-machine deployment.
• CeleryExecutor sends tasks through a broker to a persistent worker fleet that pulls and executes them.
• KubernetesExecutor creates a separate pod per task, allowing each task to select its own image and resource requests.
• Task state and coordination remain in Airflow's control plane even when execution occurs on external workers.

Airflow XCom should contain small task-to-task values or references, not large ML artifacts.

• Typical values include an object-store URI, partition identifier, model version, or compact status metadata.
• XCom rows live in the metadata layer by default, so large payloads increase database and serialization costs.
• Large datasets, checkpoints, and models belong in durable artifact storage with only their references in XCom.
• A custom object-storage XCom backend can offload serialized values, but explicit artifact contracts remain clearer.

Airflow sensors differ in whether waiting occupies a worker slot and how the next check is scheduled.

• Poke mode sleeps inside a running task and retains its worker slot between checks.
• Reschedule mode releases the worker slot and asks the scheduler to run another check later.
• A deferrable sensor hands waiting to the asynchronous triggerer and resumes the task when its trigger fires.
• Long waits generally favor deferrable or reschedule behavior, while short checks may justify poke mode.

Airflow pools and concurrency limits constrain different scopes of runnable task instances.

• A pool represents shared capacity such as GPU jobs or access to a rate-limited service, and tasks consume configured slots.
• Global parallelism caps task instances across the whole Airflow installation.
• DAG-level task concurrency limits simultaneous task instances for one DAG across active runs.
• Maximum active runs limits overlapping DagRuns and prevents a frequent schedule from multiplying downstream demand.

A Kubeflow Pipelines component specification defines a portable typed interface and the container execution needed to implement it.

• Inputs and outputs declare parameter or artifact names, types, and optional default values.
• The container section fixes the image, command, arguments, and placeholders that bind declared inputs and outputs.
• Resource requests, environment, caching options, and platform-specific settings may be attached when the component is used.
• A stable component contract lets pipeline authors replace implementation versions without changing downstream wiring.

Kubeflow Pipelines uses parameters for small values and typed artifacts for durable file-based outputs.

• Parameters carry scalar or structured control values that the backend can substitute into component arguments.
• Artifacts expose a storage path or URI plus a type and metadata instead of embedding the payload in pipeline state.
• The runtime launcher materializes input and output locations and records artifact lineage in the metadata service.
• Declared artifact types such as Dataset, Model, and Metrics make compatibility and lineage visible across components.

Kubeflow compilation converts the pipeline-building program into a static PipelineSpec package for submission.

• Component invocations become task nodes with explicit dependencies, input channels, output channels, and placeholders.
• Control constructs such as conditions, loops, and exit handling are encoded in the intermediate representation.
• The compiler validates graph wiring and type compatibility that can be determined without runtime values.
• The backend later creates a run from the package and resolves runtime parameters, artifact locations, and platform execution details.

Argo WorkflowTemplates store reusable workflow or template definitions that Workflows can reference or submit.

• A template can define steps, DAG tasks, containers, scripts, inputs, outputs, and synchronization settings.
• A Workflow may reference a namespaced WorkflowTemplate through templateRef instead of copying its implementation.
• ClusterWorkflowTemplates provide cluster-scoped reuse and therefore require deliberately managed access control.
• Versioned template names or Git-managed manifests keep callers tied to a reviewed execution contract.