Backend Developer interview questions

CAP says a distributed system can guarantee at most two of consistency, availability, and partition tolerance at once.

• Partitions are unavoidable, so the real choice is consistency vs availability during one.
• Payment ledger: PostgreSQL with synchronous replication for CP, accepting a brief read-only window on failover.
• User-preference store: Cassandra for AP, since stale data was tolerable but downtime was not.

I start from the worst-case impact of a user or business reading stale data.

• Carts and feed counts tolerate staleness; balances and inventory reservations do not.
• Strong consistency costs latency and availability, so I default to eventual consistency.
• I add compensating workflows only where business correctness demands it.
• I document the decision so the team does not silently revert to strong consistency under pressure.

CQRS separates the write model from the read model so each is optimized independently.

• Write side enforces business invariants on a normalized domain model.
• Read side materializes denormalized projections tuned for specific queries.
• It pays off when read and write loads diverge, reads need expensive aggregations, or you need multiple read shapes.
• For a simple CRUD service it is pure overhead.

Event Sourcing stores every state change as an immutable event in an append-only log instead of overwriting rows.

• Gives a full audit trail and the ability to replay events to rebuild state or new projections.
• Fits naturally with CQRS.
• Challenges: schema evolution on historical events, snapshot strategies as streams grow, thinking in events not current state.
• Reading current state needs a projection, not a simple row lookup.

I make the client send a unique idempotency key, usually a UUID, in a request header.

• The server stores the key with the result via an atomic upsert.
• A repeat request with the same key returns the original response without re-running the payment.
• Keys expire after a safe window, usually 24 hours, to limit storage growth.
• The idempotency record and the payment action share one database transaction so they never diverge.

I use Redis with a sliding window log or token bucket stored as a sorted set or a counter with expiry.

• Each instance runs a Lua script atomically so check-and-decrement is not split across round-trips.
• For very high throughput I keep a local in-process counter that syncs with Redis every 100 ms.
• That accepts a small overshoot but removes Redis from the hot path.
• I expose rate limit headers so clients can back off gracefully.

I follow the expand-and-contract pattern.

• Add the new column as nullable or with a default, then deploy code that writes to both old and new columns.
• Backfill historical rows in small batches, then drop the old column in a separate migration once all reads use the new one.
• Use CREATE INDEX CONCURRENTLY in PostgreSQL to avoid table locks.
• Treat migrations as code: in version control, run in CI, never applied manually in production.

Two-phase commit couples availability and serializes everyone behind locks during the prepare phase.

• It holds locks across all participants, so one slow service becomes a system-wide bottleneck.
• It needs a coordinator that can become a single point of failure.
• Instead I use the Saga pattern: each service runs a local transaction and publishes an event.
• If a step fails, compensating transactions roll back the previous steps, avoiding cross-service coupling and lock contention.

A Saga splits a multi-step transaction into local transactions with compensating actions.

• Choreography: each service reacts to events from the previous step, decentralized and easy to extend without changing existing code.
• Orchestration: a central coordinator calls each step in sequence, easier to trace but coupled to the orchestrator.
• I choose choreography for simple flows where the team values service autonomy.
• I choose orchestration for complex branching, retries, or a single source of truth for transaction state.

A service mesh like Istio or Linkerd injects a sidecar proxy into each pod to handle cross-cutting concerns without app code changes.

• It provides mTLS, load balancing, retries, timeouts, circuit breaking, and observability consistently.
• Cost: each proxy adds CPU and memory per pod.
• Cost: the control plane adds operational complexity and debugging through two proxy layers is much harder.
• I adopt it only when the team has the operational maturity and a shared library cannot cover these concerns.

I start with a key-value store mapping short codes to long URLs.

• Redis for hot reads backed by a durable store like DynamoDB.
• Generate short codes with a base62 counter from a dedicated ID service, or pre-generate a pool to avoid a single bottleneck.
• A CDN handles redirects at the edge for popular links, bringing reads to single-digit milliseconds.
• Rate limit the write path and push click analytics to Kafka asynchronously instead of blocking the redirect.

Backpressure signals producers that consumers are overwhelmed so queues do not grow unbounded.

• In Kafka I tune max.poll.records and batch size, and watch consumer lag as the main health signal.
• When lag crosses a threshold I alert and can add consumer instances dynamically.
• For internal reactive pipelines I use bounded queues.
• I either drop messages to a dead-letter mechanism or block the producer, depending on whether data loss is acceptable.

Vertical partitioning splits a table by columns; horizontal partitioning (sharding) splits rows across nodes.

• Vertical moves rarely used or large columns to a separate store to cut I/O on common queries.
• Sharding distributes both read and write load by a shard key.
• I use vertical partitioning to keep blobs in object storage and metadata in Postgres.
• I shard when one node can no longer handle write throughput or data volume, picking a key that spreads writes without cross-shard queries.

The approach depends on how stale the data can be.

• Mostly-static reference data: TTL-based expiration with a short enough window.
• Data that must be fresh after a write: write-through caching, updating cache and database together.
• Complex cross-entity invalidation: publish an invalidation event so every instance flushes the affected key.
• I avoid cache-aside with manual invalidation at scale because it causes read-write race conditions.

PostgreSQL gives ACID, rich queries, and simple tooling, but a single-primary write model becomes a bottleneck at scale.

• Replication lag grows under sustained high write throughput.
• Cassandra distributes writes across all nodes with no single primary, giving linear write scalability and active-active replication.
• Cassandra trade-offs: no joins, limited secondary indexes, tables designed around access patterns upfront.
• I choose Cassandra when writes exceed a single Postgres primary and query patterns are known and simple.

MVCC, multi-version concurrency control, creates a new row version per update instead of locking rows on read.

• Transaction IDs decide which version a transaction sees, so reads never block writes and vice versa.
• The cost is that old row versions accumulate until VACUUM reclaims them.
• If autovacuum cannot keep up, table bloat and transaction ID wraparound become serious risks.
• I monitor bloat and dead tuples and tune autovacuum aggressively on high-churn tables.

I find slow queries with pg_stat_statements and EXPLAIN ANALYZE before adding anything.

• I check whether the bottleneck is a sequential scan on a large table.
• I create partial indexes on filtered subsets and composite indexes ordered to match the predicate.
• I add covering indexes with INCLUDE to avoid heap fetches and always use CREATE INDEX CONCURRENTLY.
• I audit existing indexes for duplicates or unused ones via pg_stat_user_indexes, since each one slows every write.

PostgreSQL spawns a process per connection, so thousands of direct connections are very expensive in CPU and memory.

• A pooler like PgBouncer keeps a small pool of real connections and multiplexes app requests across them.
• In transaction mode it reuses a connection right after commit, serving thousands of app connections with tens of real ones.
• I run PgBouncer as a sidecar or dedicated service and tune max_client_conn to expected concurrency.
• I monitor pool wait times as a signal the pool is undersized.

Read replicas stop helping when the primary saturates or replicas serve too-stale data.

• They fail when primary write throughput saturates disk I/O or CPU.
• They fail when replication lag makes replica reads unacceptably stale, or the dataset outgrows one node.
• Next steps: vertical scaling, partitioning hot tables onto dedicated instances, a caching layer, or sharding by a distribution key.
• I exhaust vertical scaling and caching before sharding, which adds significant query and app complexity.

I identify the query from pg_stat_statements ordered by total_time or mean_exec_time.

• I run EXPLAIN (ANALYZE, BUFFERS) on a read replica or with a short statement_timeout to avoid long locks.
• I look for sequential scans, nested loops with high row estimates, or high buffer hits suggesting memory pressure.
• Common fixes: a targeted index, rewriting a correlated subquery as a join, or ANALYZE to refresh stale statistics.
• I stage any index creation or rewrite in a non-production environment first.