Growth Marketer interview questions

A sequential experiment permits planned interim decisions, while a fixed-horizon test is calibrated for one analysis at the target sample.

• A valid sequential design defines when results may be checked and which boundary allows stopping.
• Its confidence sequence or test statistic remains calibrated across those planned looks.
• The flexibility usually costs some power or requires more evidence than a one-time fixed-horizon analysis.

Alpha spending allocates a test's false-positive budget across interim analyses.

• An early look receives only part of the total alpha, so its stopping boundary is usually strict.
• Spending functions such as O'Brien-Fleming preserve more alpha for later looks, while Pocock-style rules spread it more evenly.
• The schedule and spending rule must be chosen before outcomes are inspected.

Each additional hypothesis creates another chance to find a random win, so the family-wise error exceeds the per-test threshold.

• With 20 independent null tests at alpha 0.05, the chance of at least one false positive is about 64 percent.
• Bonferroni controls family-wise error by dividing alpha across tests, while Holm is usually less conservative.
• The correction family should match the decision family, such as all treatment arms compared for one launch.

False discovery rate is useful when screening many hypotheses and tolerating a controlled share of false leads is acceptable.

• Benjamini-Hochberg controls the expected proportion of false discoveries under independence or suitable positive dependence; arbitrary dependence needs a more conservative guarantee such as Benjamini-Yekutieli.
• It is less strict than family-wise control, so it retains more power in broad creative or audience screens.
• A high-stakes ship decision still usually needs a confirmatory test with tighter error control.

A Bayesian result updates a prior with observed data to produce a posterior distribution for the treatment effect.

• A 95 percent credible interval directly contains 95 percent of the posterior probability under the model.
• The posterior can answer questions such as the probability that lift exceeds a two-point practical threshold.
• The result is conditional on the likelihood, prior, and data-quality assumptions, not free from assumptions.

Prior-sensitivity analysis shows whether the decision depends heavily on assumptions made before the test.

• A weakly informative prior can stabilize sparse data without dominating a large sample.
• An optimistic historical prior should be discounted when the audience, surface, or metric definition has changed.
• Comparing skeptical, neutral, and informed priors reveals whether the posterior conclusion is robust.

A sound Bayesian decision rule weighs posterior outcomes against the costs of shipping, delaying, and causing harm.

• A 70 percent probability of a tiny lift may be enough for a reversible copy change but not for an expensive rebuild.
• Expected loss combines each possible effect with the business consequence of the corresponding decision.
• A practical threshold prevents a high probability of a negligible effect from being called valuable.

A bandit adapts traffic toward better-performing arms during learning, while a standard A/B test preserves a planned allocation for cleaner effect estimation.

• Bandits suit repeated, short-lived choices such as rotating offers when reward arrives quickly.
• Naive analysis can be invalid when outcome-adaptive allocation, stopping, or time drift is ignored, so assignment probabilities and adaptive-valid inference must be preserved.
• A fixed randomized test is usually better when the goal is an interpretable causal estimate or delayed retention outcome.

Regret is the cumulative reward lost by the bandit's choices compared with always selecting the best arm in hindsight.

• Pulling weak arms creates short-term regret but supplies information needed to identify stronger arms.
• Cumulative regret captures the full cost over time, not only which arm wins at the end.
• Low-regret algorithms balance learning speed with the value sacrificed during exploration.

Both favor promising arms while preserving exploration through uncertainty, but they express uncertainty differently.

• Thompson sampling draws a reward estimate from each arm's posterior and selects the highest draw.
• An upper confidence bound method chooses the arm with the strongest optimistic estimate based on its mean and uncertainty bonus.
• Sparse or newly added arms receive exploration because their uncertainty is wider than that of well-measured arms.

Network effects create interference because one user's treatment can change another user's outcome.

• A referral incentive shown to one member can affect signups among control-group friends.
• This violates the assumption that each unit's outcome depends only on its own assigned treatment.
• Contamination can dilute, reverse, or otherwise distort the estimated user-level treatment effect.

Cluster randomization assigns connected units together so treatment is less likely to spill between test groups.

• Clusters may be households, companies, schools, or geographic areas depending on where interaction occurs.
• Users inside a cluster are correlated, so effective sample size depends more on cluster count than raw user count.
• Analysis must use cluster-aware standard errors and check balance across the randomized clusters.

Guardrails should protect a plausible failure mode of the treatment and have a predefined action threshold.

• A faster signup flow may guard payment fraud or qualified activation rather than monitor every dashboard metric.
• A non-inferiority margin states how much deterioration is tolerable before the treatment is rejected.
• Too many noisy guardrails create multiplicity and make almost every test look ambiguous.

A long-term holdout keeps a stable untreated group to measure cumulative effects that short experiments miss.

• It can reveal delayed churn, channel cannibalization, or repeated-message fatigue after an early conversion lift.
• Eligibility and assignment must remain stable so users do not drift between holdout and treatment over time.
• The holdout has an opportunity cost because part of the audience does not receive improvements.

Exposure should mark the first point where a unit can experience the treatment, not a background assignment that may never reach the user.

• Logging every assigned user as exposed dilutes the estimate when many never load or enter the changed surface.
• Logging only after engagement is also biased because the treatment may influence who reaches that later event.
• The record needs the experiment, variant, unit, and time at the treatment point, while repeated evaluations are deduplicated by the analysis unit.
• Assignment can remain a separate record for intention-to-treat analysis and debugging.

Post-randomization filtering creates SRM when treatment changes which assigned users remain in the analyzed sample.

• Intention-to-treat analysis preserves randomization by keeping users in their assigned groups whether or not they complete the experience.
• A triggered analysis is credible only when the trigger occurs before treatment can affect it and is logged symmetrically across variants.
• Balanced assignment counts can coexist with imbalanced exposure counts when a variant-specific load failure removes treated users.

CUPED uses a pre-experiment covariate correlated with the outcome to remove predictable variation from the metric.

• Prior-week purchase frequency can improve precision for a current purchase metric when measured before assignment.
• A stronger correlation yields a larger variance reduction and can shorten the sample needed for the same MDE.
• The covariate must not be affected by treatment, and the adjusted estimator must preserve unbiasedness.

Novelty changes behavior because an experience is new, while carryover lets an earlier treatment influence later observations.

• A redesigned dashboard may attract temporary exploration that inflates first-week engagement.
• A messaging treatment can keep affecting users after they switch groups, especially if it changed habits or expectations.
• Full business cycles, stable assignment, and washout periods where appropriate help separate durable effects from timing artifacts.

A stock is a quantity measured at a point in time, while a flow adds to or removes from that quantity over an interval.

• Active customers are a stock, while new, churned, and resurrected customers per month are flows.
• The ending stock equals the starting stock plus inflows minus outflows over the same period.
• Mixing a monthly flow with a daily stock produces a model with inconsistent units and misleading rates.

Growth accounting explains the active base using retained, new, resurrected, and churned users across two periods.

• The ending base equals retained plus new plus resurrected users; churned users are not active at the end.
• The prior base equals retained plus users who churned during the period.
• Net change therefore equals new plus resurrected minus churned users when the categories are mutually exclusive.