Data Scientist interview questions

Random forest emphasizes robustness, while gradient boosting usually emphasizes maximum tabular accuracy.

• Random forest trains deep trees independently on bootstrapped rows and random feature subsets, then averages them to reduce variance.
• Gradient boosting adds shallow trees sequentially to reduce the current loss, lowering bias but increasing sensitivity to learning rate, tree count, and noise.
• I use boosting as the main performance candidate and random forest as a strong baseline or when I need little tuning, parallel training, and stable importance estimates.

I reduce XGBoost overfitting by controlling step size, tree capacity, randomness, and explicit regularization.

• Lower learning_rate, increase the tree count, and use early stopping on untouched validation data so many small updates replace a few aggressive ones.
• Reduce max_depth or increase min_child_weight to prevent trees from memorizing rare patterns, then use subsample and colsample to decorrelate them.
• Raise gamma for stricter split gains and lambda or alpha for leaf-weight penalties, while judging every change on data the model never trains on.

XGBoost and LightGBM often reach similar tuned quality, but their growth strategy and scaling behavior differ.

• LightGBM grows leaf-wise by splitting the leaf with the largest loss reduction, which is efficient but can create deep uneven branches that overfit small data.
• Classic XGBoost grows level-wise and is more balanced, while LightGBM's histogram methods, sampling, and native categorical support make it faster on large datasets.
• I favor LightGBM for millions of rows and rapid iteration, controlling num_leaves with max_depth, and otherwise choose by ecosystem and measured validation results.

Gradient boosting builds an additive model by performing gradient descent in function space.

• It starts with a constant prediction and computes each sample's loss gradient with respect to the ensemble's current prediction.
• The next tree fits those gradients, its contribution is multiplied by the learning rate, and the result is added to the ensemble before repeating.
• With squared error the gradients are residuals, while the same machinery supports other differentiable objectives such as logistic, quantile, and ranking losses.

Default boosted-tree importance can reflect training-data bias rather than genuine predictive value.

• Split-count and gain scores favor features with many split points, while correlated features divide credit arbitrarily between near-duplicates.
• I rank features with permutation importance on held-out data and use grouped permutation when correlated variables should be evaluated together.
• SHAP adds direction and per-sample attribution, but the decisive check is retraining without a suspicious feature and measuring the validation change.

SHAP explains both the direction and size of feature contributions for individual predictions.

• Each prediction is decomposed relative to a baseline, showing which features pushed a specific score up or down rather than only ranking them globally.
• Aggregated SHAP values provide global views, dependence plots reveal changing effects and interactions, and explanation clusters expose distinct subpopulations.
• With stakeholders I use case-level narratives and plausibility checks, while stating clearly that SHAP explains model behavior, not causal effects.

SVM remains effective for small or medium, high-dimensional datasets when its geometric requirements are respected.

• A linear SVM is especially strong for sparse text with thousands of TF-IDF features, while kernels can model nonlinear boundaries through implicit mappings.
• Feature scaling is mandatory because margins and RBF distances are geometric, and one large-scale feature can otherwise dominate the model.
• Kernel SVMs scale poorly beyond a few hundred thousand samples, probabilities need calibration, and C and gamma must be tuned together.

Meaningless k-means segments usually indicate violated geometry assumptions or an unsuitable feature space.

• K-means assumes roughly spherical clusters of similar size and density under Euclidean distance, and it always returns k groups even if no clusters exist.
• I verify scaling, inspect PCA or UMAP projections, and compare silhouette scores across k values while allowing for the possibility of one continuous cloud.
• If the groups lack business meaning, I rebuild features around relevant behaviors before tuning k, because a different k cannot repair irrelevant inputs.

DBSCAN gains flexible cluster shapes and explicit noise handling, but depends heavily on meaningful local density.

• It groups points having at least min_samples neighbors within eps, finds arbitrary shapes, requires no preset k, and leaves sparse points as noise.
• One eps cannot represent clusters with very different densities, distance concentration breaks density in high dimensions, and eps itself needs diagnostics such as a k-distance elbow.
• I use DBSCAN for low-dimensional spatial or noise-heavy data, and prefer k-means or Gaussian mixtures for scalable blob-like clusters, often after dimensionality reduction.

L1 creates sparse models, while L2 usually provides smoother and more stable shrinkage.

• L1 has a constant penalty gradient and a diamond-shaped constraint with corners on the axes, so small coefficients can become exactly zero.
• L1 suits feature selection and sparse interpretation, whereas L2 spreads weight across correlated features and is often the safer predictive default.
• Elastic net combines both for correlated groups when sparsity matters, with standardized features and penalty strength selected by cross-validation.

Logistic regression can beat boosting when data, representation, or operational requirements favor a linear model.

• It performs well on small datasets and wide sparse inputs such as text or heavy one-hot encoding, where boosting may overfit.
• It is easy to deploy, offers an auditable formula, and often gives better calibrated probabilities without post-processing, making it a necessary baseline.
• Its raw coefficients are log-odds, require scaling for comparison, become unstable with correlated features, and describe association rather than causation.

Tree and linear models need different preprocessing because they represent relationships differently.

• Trees split by order-preserving thresholds, so scaling, normalization, and monotonic log transforms do not change their splits and are usually wasted for XGBoost.
• Linear and distance-based models need scaling, often benefit from log transforms for skew, and require explicit bins, splines, or interactions for nonlinear structure.
• For trees I focus on categorical encoding, missing-value semantics, and domain features such as ratios that would otherwise require many splits.

K-nearest-neighbors fails in high dimensions because geometric proximity stops carrying useful contrast.

• Distances concentrate, so the relative gap between nearest and farthest neighbors shrinks and the nearest point is barely more meaningful than a random one.
• High-dimensional volume also makes a fixed dataset sparse, while irrelevant dimensions disproportionately dilute signal in the distance metric.
• For kNN, clustering, and RBF methods I reduce dimensions, learn embeddings or a metric, or switch to models such as trees that implicitly select useful features.

A model is miscalibrated when its predicted probabilities do not match observed event frequencies.

• If only 60 percent of cases scored 0.9 are positive, a reliability diagram exposes the gap from the expected 90 percent.
• Forest averaging often avoids extremes, boosting can become overconfident, and class-imbalance resampling changes the base rate embedded in scores.
• I calibrate on held-out data with Platt scaling for smooth corrections or isotonic regression with more data, especially when probabilities drive value, risk, or threshold decisions.

Stacking is worthwhile only when diverse strong models add enough quality to justify operational complexity.

• Different algorithms, feature views, or training subsets can decorrelate errors, letting a meta-model gain roughly one to three percent over the best base model.
• That gain may decide a competition, but in production it must cover multiplied latency, serving cost, and maintenance versus one tuned boosting model with better features.
• Meta-features must be out-of-fold predictions from rows unseen by each base model, or leakage teaches the meta-model to trust overfit scores.

A ten-thousand-level categorical feature needs compact encoding with explicit protection against leakage and rare levels.

• One-hot encoding creates ten thousand sparse columns, wastes memory, and leaves too little data for many tree splits.
• Better choices are native categorical handling in LightGBM or CatBoost, out-of-fold smoothed target encoding, or leak-free hashing and frequency encoding as simpler fallbacks.
• I prefer native support, otherwise group the rare tail into an other bucket and use smoothed out-of-fold target statistics so categories cannot memorize labels.

Missing-value handling should match the model and preserve informative absence without leaking collection artifacts.

• XGBoost and LightGBM can receive NaN directly and learn a default split direction, often using the missingness pattern better than imputation.
• Linear models and neural networks need explicit imputation, such as medians for skewed numbers or a separate categorical value, plus a binary missing indicator.
• Missingness can be predictive when nonresponse or sensor silence carries meaning, but collection decisions linked to the outcome can create leakage that disappears in deployment.

Outliers require different treatment by model family, and provenance should determine whether they are corrected or modeled.

• Squared-loss regression, gradient updates, k-means centroids, and fitted scalers can be dominated or distorted by extreme observations.
• Trees are robust to extreme feature magnitudes because splits use order, but extreme regression targets still distort leaf values under squared loss.
• I fix or remove proven errors, model real extremes with MAE or Huber loss, log transforms, or RobustScaler, and cap values only when the business explicitly ignores the tail.

Multicollinearity mainly threatens coefficient interpretation, not predictive performance.

• Correlated predictors make linear coefficients unstable, inflate standard errors, and can flip signs across samples, with VIF serving as a common diagnostic.
• Prediction quality often changes little, and trees tolerate correlation, although their importance scores arbitrarily divide credit within the correlated group.
• For inference I remove or combine variables or use ridge regularization; for prediction I may do nothing, and for interpretation I evaluate correlated features as groups.

For a new tabular problem, I move from a trivial baseline to linear models and then to tuned gradient boosting.

• A majority-class or mean predictor defines the floor, while logistic or linear regression reveals whether simple structure already explains most of the signal.
• Boosting is the main performance candidate because it handles mixed scales, nonlinear thresholds, interactions, and missing values well on modest tabular datasets.
• I prefer linear models for tiny data or strict interpretability, neural methods for text, images, or sequences, and require any complex model to beat the baseline honestly.