Kubernetes Engineer interview questions
100 real questions with model answers and explanations for Senior candidates.
See a Kubernetes Engineer resume example →Practice with flashcards
Spaced repetition · Hunter Pass
Questions
I would run three control-plane nodes and three stacked etcd members, one per rack, behind two load balancers with a virtual IP.
- Three etcd members tolerate one rack loss; five add write latency without improving the stated one-rack requirement.
- Scheduler and controller-manager run three replicas with leader election, while all three API servers accept traffic.
- I reserve CPU and memory for control-plane components and test rack isolation plus load-balancer failover before launch.
Why interviewers ask this: The interviewer is checking whether the candidate turns an availability target into a concrete quorum, placement, and failover design.
I would choose five members because a five-member quorum survives two failures, while three survives only one.
- I would place them 2-2-1 across the three zones and verify that losing either two-member zone still leaves three votes.
- Cross-zone round-trip time must stay low and stable, ideally below 10 ms, because every committed write waits for a majority.
- I would not use six members: it still tolerates only two failures and adds another replication target.
Why interviewers ask this: A strong answer gets the quorum math right and also accounts for asymmetric zone placement and write latency.
I would control object churn first, then size and measure etcd rather than treating disk capacity as the only limit.
- I use dedicated low-latency SSDs and alert on fsync and WAL p99 latency before they approach the 100 ms warning range.
- I reduce noisy status updates, large Secrets and ConfigMaps, and controllers that relist instead of watching.
- I compact revisions, defragment members one at a time, and load-test the rollout rate against the exact Kubernetes and etcd versions.
Why interviewers ask this: The interviewer wants practical understanding of etcd bottlenecks across disk latency, object churn, and maintenance.
I would take encrypted snapshots every 15 minutes and prove a full control-plane restore completes inside 60 minutes.
- Snapshots go to a separate account or region with retention and integrity checks, not to a volume in the same cluster.
- The runbook restores a new logical etcd cluster, updates API server endpoints and certificates, then checks controllers and workloads.
- A scheduled restore test records snapshot age, restore duration, missing external resources, and the actual achieved RPO and RTO.
Why interviewers ask this: The interviewer checks whether backup frequency, restore mechanics, and measured recovery objectives form one usable design.
I would remove the relist pattern before adding API server replicas because it multiplies work across every object.
- Controllers should use shared informers, watches, pagination, and narrow label or field selectors instead of repeated full LIST calls.
- I set API Priority and Fairness so kubelet, leader-election, and system-controller traffic keeps seats under tenant load.
- I track request rate, inflight requests, watch-cache misses, response size, and etcd latency while replaying the expected fleet load.
Why interviewers ask this: A strong answer fixes the request pattern and protects critical traffic instead of relying only on horizontal scaling.
I would use shared regional clusters for ordinary teams and dedicated PCI clusters for the twelve regulated workloads.
- Namespaces, RBAC, quotas, and NetworkPolicy provide soft multi-tenancy for teams inside the same trust boundary.
- PCI workloads get separate control planes, node identities, audit trails, and upgrade windows because namespace isolation is not a hard security boundary.
- I standardize both classes with the same Cluster API templates and policy bundle so extra clusters do not become snowflakes.
Why interviewers ask this: The interviewer evaluates whether cluster boundaries follow trust and compliance requirements rather than team count alone.
I would run one autonomous cluster per region and keep each control plane and etcd quorum inside its region.
- Global DNS or an anycast edge sends users to the nearest healthy regional ingress.
- Workload configuration is replicated through GitOps, but live API and etcd traffic never crosses the Atlantic.
- Stateful services need an explicit replication and failover design because Kubernetes federation does not solve cross-region data consistency.
Why interviewers ask this: A strong answer keeps control-plane failure domains local and separates workload placement from data replication.
I would reserve enough capacity in any two zones to carry the critical load after losing the third.
- Node pools span all three zones, but minimum capacity is calculated at roughly 50% headroom across the surviving pair, not equal node count alone.
- Critical Deployments use topologySpreadConstraints with maxSkew 1, plus Pod anti-affinity where duplicate placement is unacceptable.
- PodDisruptionBudgets protect voluntary drains, while readiness and failover tests prove the application survives an involuntary zone loss.
Why interviewers ask this: The interviewer checks whether topology, spare capacity, and workload constraints actually satisfy the failure requirement.
I would define a sharding boundary now and validate the intended per-cluster ceiling with load tests.
- I shard by region and trust domain first because those boundaries also reduce failure and access blast radius.
- I test API request rate, watch count, EndpointSlice volume, scheduler latency, and etcd database growth at projected churn.
- I avoid arbitrary 500-node shards because every extra cluster duplicates add-ons, upgrades, capacity buffers, and operating cost.
Why interviewers ask this: A strong answer treats published Kubernetes limits as ceilings and derives cluster size from measured control-plane load and blast radius.
I would give the entire admission chain a strict latency budget and remove noncritical synchronous work.
- With a 500 ms API target, I budget roughly 100 ms p99 per webhook and leave the remainder for authentication, storage, and network time.
- Each webhook runs at least three replicas across zones with readiness probes, a PodDisruptionBudget, and no dependency on the workload it admits.
- Security checks fail closed; advisory mutation may fail open, but every failure policy and timeout is tested during a webhook outage.
Why interviewers ask this: The interviewer wants a concrete admission design that balances policy enforcement with control-plane availability.
I would use one namespace per application and environment, owned through team labels rather than one giant namespace per team.
- Names like payments-api-prod make ownership and environment explicit, while labels drive quotas, policies, cost reports, and automation.
- A namespace template creates the RoleBindings, ResourceQuota, LimitRange, default-deny policies, and service account baseline.
- I cap namespace count only after measuring control-plane load; 2,700 namespaces are manageable if controllers watch efficiently.
Why interviewers ask this: The interviewer checks whether namespace boundaries support ownership, policy automation, and scale without becoming an arbitrary hierarchy.
I would bind a reusable ClusterRole through a RoleBinding in that namespace and exclude privilege-escalation surfaces.
- The role can manage namespaced workloads, Services, ConfigMaps, and its own application Secrets.
- It cannot create RoleBindings to broader roles, impersonate users, modify admission webhooks or CRDs, or access Nodes and PersistentVolumes.
- Workloads use separate service accounts with narrower Roles, and audit rules flag attempts to change RBAC or exec into protected Pods.
Why interviewers ask this: A strong answer knows that namespaced admin access still needs explicit protection from RBAC escalation and cluster-scoped resources.
I would guarantee less than the peak request and make the burst visible rather than reserving 40% of the cluster permanently.
- I might set requests.cpu quota to 200 cores and limits.cpu to 400 after checking the team's measured p95 and business priority.
- Namespace quotas also cap memory requests, object counts, PVC capacity, and LoadBalancer Services to protect non-CPU control planes.
- Prometheus alerts at 80% quota and a reviewed increase path prevent teams from discovering the limit during a release.
Why interviewers ask this: The interviewer evaluates whether quotas encode measured guarantees, burst policy, and scarce non-CPU resources.
I would introduce conservative defaults per workload class, not one large default that hides bad sizing.
- For ordinary web Pods I might start at 100m CPU and 128 MiB memory requests, with limits of 1 CPU and 512 MiB.
- Batch and JVM namespaces get different templates because the same defaults would make scheduling and OOM behavior misleading.
- I first audit and warn, then enforce after teams set explicit values; otherwise defaults can silently cause throttling or OOMKills.
Why interviewers ask this: A strong answer uses LimitRange as a safety net while acknowledging that generic defaults can create new production failures.
I would generate observed allow rules first, then enforce namespace cohorts rather than flipping the whole fleet at once.
- Every namespace receives default-deny ingress and egress plus explicit DNS access to kube-dns on TCP and UDP 53.
- Platform namespaces publish stable labels for ingress, metrics scraping, tracing, and approved shared services.
- I inspect Cilium Hubble or Calico flow logs for two weeks before enforcement, and each cohort has a tested rollback commit.
Why interviewers ask this: The interviewer checks whether the candidate can make a restrictive network baseline deployable across real shared-cluster dependencies.
The platform team would own the Gateway and listeners, while application teams own HTTPRoutes in their namespaces.
- allowedRoutes restricts attachment to namespaces carrying an approved label, so a tenant cannot claim every hostname.
- ReferenceGrant is required for intentional cross-namespace backend or certificate references rather than granting broad access.
- Admission policy validates hostname ownership and TLS requirements before a route reaches the shared gateway.
Why interviewers ask this: A strong answer uses Gateway API's resource boundaries to separate infrastructure ownership from tenant routing.
I would ship one versioned Kyverno bundle through GitOps and move each rule from Audit to Enforce after measuring violations.
- The bundle checks runAsNonRoot, blocks privileged and host namespace access, verifies image signatures, and pins images by digest.
- Exemptions require a namespace, owner, reason, and expiry date instead of a permanent wildcard.
- I canary one cluster, watch admission p99 and rejection counts, then promote in waves with a tested rollback version.
Why interviewers ask this: The interviewer evaluates policy coverage, exception hygiene, performance, and staged enforcement at fleet scale.
I would bound both active compute and Kubernetes object churn for that tenant.
- ResourceQuota caps active Jobs, Pods, CPU, and memory, while TTLAfterFinished removes completed Jobs after a short retention such as one hour.
- Kueue admits batch work against a tenant quota instead of letting thousands of Pods enter Pending at once.
- API Priority and Fairness gives the tenant a limited request share so kubelet and system-controller traffic remains responsive.
Why interviewers ask this: A strong answer protects compute, object count, and API capacity instead of treating noisy-neighbor control as CPU quota only.
It should create a complete tenancy boundary, not just the Namespace object.
- The workflow assigns owner and cost labels, RoleBindings, workload service accounts, ResourceQuota, and LimitRange.
- It installs default-deny NetworkPolicies, Pod Security Admission labels, approved registry policy, and baseline observability hooks.
- A conformance Job verifies DNS, image pulls, metrics, policy rejection, and quota before marking onboarding complete within the target 15 minutes.
Why interviewers ask this: The interviewer checks whether namespace provisioning is a tested platform contract rather than a YAML creation step.
No, I would place hostile code in a separate cluster or stronger sandbox boundary rather than rely on namespaces alone.
- Namespaces share the kernel, kubelet, CNI dataplane, and control plane, so a container escape crosses the intended tenant boundary.
- If density matters, I add gVisor or Kata Containers, dedicated tainted nodes, separate identities, and strict egress, but these are additional controls.
- Internal multi-tenancy can remain namespace-based because its threat model is accidental interference, not mutually hostile execution.
Why interviewers ask this: A strong answer distinguishes soft organizational tenancy from hard isolation against untrusted code.
Locked questions
- 21
Choose a CNI for a 500-node EKS cluster that needs NetworkPolicy, flow visibility, and no kube-proxy. What would you pick and why?
proxynetworkingcluster - 22
For 300 on-prem nodes across two data centers, would you use an overlay or native routed Pod network?
workloads - 23
You need portable L3 and L4 policy on mixed Linux versions, but not kube-proxy replacement. Would you choose Calico or Cilium?
proxy - 24
A cluster has 12,000 Services and 80,000 endpoints. Which service dataplane would you test: iptables, IPVS, or eBPF?
clusterendpoints - 25
Plan Pod IP capacity for 1,000 nodes with up to 110 Pods per node and 30% growth headroom.
capacityworkloads - 26
Nodes use a 1500-byte underlay and VXLAN. What Pod MTU would you configure, and how would you prove it?
workloadsconfig - 27
How would you validate that 10,000 NetworkPolicies do not make rollouts unsafe?
validation - 28
A regulated tenant may reach only 40 approved SaaS domains. How would you implement Kubernetes egress control?
kubernetescloud - 29
Would you enable dual-stack IPv4 and IPv6 on a new 200-node cluster if only 10% of services need IPv6?
cluster - 30
Connect ten clusters with overlapping Pod CIDRs while keeping failures isolated. What network model would you use?
workloadscluster - 31
A web API runs 20 replicas at 500m CPU request each and should scale at 60% utilization. Traffic doubles in five minutes. How would you configure HPA?
replicationapiconfig - 32
CPU is flat, but an API must keep queueing below 50 requests per Pod. Which HPA metric would you use?
workloadsapimonitoring - 33
An HPA alternates between 30 and 80 replicas every few minutes. What settings and signals would you change?
replication - 34
A JVM service is over-requested by 3x but uses HPA on CPU. How would you introduce VPA without creating conflicting loops?
- 35
A worker processes 20 messages per second and the queue may jump to 120,000 messages. Design KEDA scaling to drain it within 10 minutes.
scalingdesignconcurrency - 36
For EKS with unpredictable instance shapes and 90-second scale-up SLO, would you choose Cluster Autoscaler or Karpenter?
scalingslocluster - 37
Nodes take four minutes to become Ready, but the workload SLO allows only 60 seconds of scale-up delay. How would you close the gap?
slo - 38
Cluster utilization is 25%, but teams claim requests cannot be reduced safely. How would you produce new request values?
cluster - 39
Design Karpenter capacity for a service that needs 99.95% availability but can save money on interruptible workloads.
designcapacityautoscaling - 40
A StatefulSet scales from 6 to 20 replicas, but each new volume takes 90 seconds to provision. How would you set autoscaling expectations?
scalingworkloadsreplication - 41
Design an operator that provisions one external database per Database CR without duplicating databases after retries.
designoperatorsdatabase - 42
A v1 CRD stores replicas as an integer, while v2 replaces it with a scaling policy object. How would you migrate without breaking clients?
replicationscaling - 43
An operator-managed cloud bucket must be deleted when its CR is deleted. How would you prevent both leaks and stuck deletion?
problem-solvingoperators - 44
Your operator manages 20,000 resources, but the provider allows 100 API calls per second. How would you shape reconciliation?
reactapioperators - 45
A team wants to store 50,000 high-frequency sensor updates per second in CRDs. Would you approve it?
- 46
A StatefulSet spans three zones and uses zonal block volumes. Which StorageClass binding mode would you choose?
workloadsstorage - 47
Would you run a three-member PostgreSQL cluster as a StatefulSet across three zones? What does Kubernetes solve and what remains?
workloadskubernetescluster - 48
A database needs 15,000 IOPS and p99 storage latency below 5 ms. How would you select and verify a CSI-backed volume class?
databaselatency - 49
A 2 TiB PVC is 85% full and grows 20 GiB per day. What expansion and backup plan would you execute?
backups - 50
Design storage recovery for a stateful service with a 5-minute RPO and 30-minute RTO after losing an entire zone.
design - 51
One of 60 worker nodes fails at 8,000 requests per second, taking six of 48 API Pods with it. What do you do in the first 15 minutes?
api - 52
A zone containing 35% of cluster capacity disappears, and checkout errors rise from 0.2% to 4%. Walk me through the response.
capacitycluster - 53
Twelve of 200 nodes become NotReady for 90 seconds every eight minutes after a node-image rollout. Do you drain them?
- 54
300 Pods are evicted in ten minutes as five nodes report MemoryPressure. How do you stop the eviction cascade?
memory - 55
A Java Pod with a 4 GiB memory limit is OOMKilled six times per hour, but its normal working set is 3.6 GiB. What do you change?
workloadsmemory - 56
Forty nodes enter DiskPressure because application logs consume 85% of ephemeral storage. What is your containment plan?
- 57
The API server returns 429 to kubelets while a CI system sends 12,000 LIST requests per second. How do you recover?
system-designcontrol-planeapi - 58
API p99 jumps from 180 ms to 2.5 seconds after a validating webhook release. How do you prove and roll it back?
webhooksvalidation - 59
etcd is at 7.6 GiB of an 8 GiB quota and write p99 is 140 ms. What sequence do you follow?
control-plane - 60
CoreDNS reaches 70,000 queries per second and 18% of lookups time out after a mobile release. What do you do?
queries - 61
After upgrading Cilium on 20% of nodes, cross-node packet loss reaches 5% while same-node traffic is healthy. What do you inspect and roll back?
rollback - 62
Only eight of 150 nodes show intermittent Service timeouts, and conntrack usage is above 95% on those nodes. What is your fix?
resilience - 63
600 Pods are Pending during a sale, but the node autoscaler adds nothing. How do you classify the blocker in five minutes?
communicationscalingautoscaling - 64
A tenant has a 4-core CPU limit, shows 55% throttling, and misses latency SLO even though the cluster has 300 idle cores. How do you fix it?
sloclusterlatency - 65
One tenant requests 20 GiB but uses 180 GiB, causing 90 Pods from other teams to be evicted. What do you do now and after recovery?
- 66
An ML team submits 800 GPU Pods and starves a latency-sensitive inference service that needs 40 GPUs. How do you restore priority?
latency - 67
A custom controller enters a retry loop and drives API traffic from 2,000 to 18,000 requests per second. How do you stop it safely?
zero-to-oneapiresilience - 68
Monthly cluster cost rises from $180,000 to $310,000 in seven days with no traffic growth. How do you find and stop the increase?
cluster - 69
Karpenter consolidation saves 28%, but it replaces 120 nodes per day and increases checkout errors during drains. What would you change?
autoscaling - 70
Spot capacity reaches 60% of a cluster, then a market interruption removes 25% of nodes in six minutes. How do you recover and revise the mix?
capacitycluster - 71
Cross-zone network charges jump by $90,000 per month after a topology change. How do you reduce them without concentrating replicas in one zone?
replication - 72
An audit finds 600 unattached volumes and 90 unused load balancers costing $42,000 per month. How do you clean them up safely?
load-balancing - 73
A production Pod with access to 3 Secrets starts mining cryptocurrency and sends traffic to an unknown IP. What are your first containment actions?
workloadsconfigurationsecrets - 74
A database password appears in centralized logs for 36 hours, and 14 Pods could use it. What response do you coordinate?
databasepasswords - 75
You discover that 220 developers can create RoleBindings to cluster-admin. What do you do in the next hour?
cluster - 76
Tetragon shows a privileged process modifying host files on one node. How do you handle the node and its 70 resident Pods?
soft-skillsconcurrency - 77
A signed production image digest is linked to a compromised build runner and is running in 11 clusters. What is your response?
cluster - 78
Falco reports shell execution in 900 Pods after a legitimate support tool rollout. How do you avoid both alert blindness and 900 pages?
runtime-securityalerting - 79
A default-deny NetworkPolicy rollout blocks payment traffic in 40 namespaces. Do you revert globally?
namespaceskubernetes - 80
A Kyverno release rejects 40% of Pod creates and admission p99 reaches 900 ms. How do you recover delivery?
workloads - 81
After upgrading a canary cluster, 30 applications fail because a deprecated API is no longer served. The control plane cannot be downgraded. What now?
deployment-strategiescontrol-planecluster - 82
A new node image causes 8% of replacement nodes to fail CNI initialization during an upgrade. How do you roll back without exhausting capacity?
capacitynetworkingrollback - 83
During a CNI migration, 5% of canary Pods lose DNS and egress. How do you decide between repair and rollback?
dnsdeployment-strategiesnetworking - 84
A CSI migration leaves 12 of 200 StatefulSet Pods unable to mount volumes. What is your rollback path?
migrationsrollbackworkloads - 85
A Karpenter migration cuts node cost 22% but doubles Pending time to 140 seconds and violates the scale-up SLO. Do you keep it?
migrationssloautoscaling - 86
Traffic cutover to a new cluster raises checkout errors from 0.3% to 2% at 25% traffic. What do you roll back first?
clusterrollback - 87
A shared cluster with 120 teams has had three tenant-caused outages in two months. Would you split it now?
cluster - 88
A company serves three regions, requires PCI isolation, and has a six-person platform team. How many cluster classes would you operate?
cluster - 89
Eighty services need mTLS, but the checkout path has only 2 ms of latency budget and the platform team has four engineers. Would you deploy a service mesh?
service-meshdeploymentlatency - 90
An Istio rollout adds 40 ms to checkout p99 and increases Pod memory by 18%. How do you back it out?
workloadsmemory - 91
A region serving 40% of traffic fails, while database replicas in the backup region lag by 90 seconds. Do you fail over writes?
databasereplicationbackups - 92
An emergency kubectl patch stopped an outage, but Argo CD will revert it in three minutes. What do you do?
kubernetesgitopskubectl - 93
The platform team receives 180 pages per week, but only 12 require action. What do you change first?
- 94
A cluster outage affects 45% of customers, and 18 engineers join the call in ten minutes. How do you command it?
joinscluster - 95
A bad node rollout caused a 52-minute outage, and the initial review blames the engineer who clicked deploy. How do you redirect it?
deployment - 96
During a DNS incident, a mid-level engineer wants to restart all 12 CoreDNS Pods at once. How do you mentor without taking over?
mentoringincidentsdns - 97
A teammate proposes deleting a blocking PodDisruptionBudget to finish a 200-node upgrade tonight. How do you handle the decision?
soft-skills - 98
Two senior engineers disagree on Cilium versus Istio for mTLS across 70 services, and the decision is blocking a migration. How do you resolve it?
conflictmigrationsmtls - 99
The same tenant has caused three API saturation incidents by creating 25,000 Jobs per hour. Informal warnings failed. What do you enforce?
incidentsapi - 100
You can fund either self-service cluster upgrades or a new GPU platform this quarter. Upgrades consume 220 engineer-hours monthly and caused two outages; GPU demand is from three teams. What do you choose?
cluster