For temporal models, data leakage has a simple definition: using information from the future to predict the present. Any feature that, at time t, encodes something you could only know at t+1 or later is leakage.
Every practitioner who has been burned by it knows the pattern. Training metrics look great. Validation looks great. Then you deploy, and performance collapses — because in production the future genuinely isn’t available, and the model was quietly leaning on it the whole time. Worse than the collapse is what it does to your evaluation: every number you produced after training is wrong. You weren’t measuring what you thought you were measuring.
What still surprises me, after years of building these pipelines, is how many different ways leakage sneaks in. It is rarely one obvious mistake. It hides in feature construction, in a shift operator pointing the wrong way, in a column name, in the architecture of the model itself. Here are the ones I keep running into, and the checks that catch them.
1. Features that quietly reach into the future
The most common form: a feature meant to describe time t is actually computed using information from t+1 or later. Sometimes it’s a join that pulls in a value stamped in the future. Sometimes it’s an aggregate that spans the wrong window. The feature looks perfectly reasonable in a finished dataframe, because by then every row already has its neighbors.
The check: walk the dataframe one time step at a time and ask whether you could build every feature using only rows at or before the current step. If a feature needs a row that hasn’t “happened” yet, it’s leakage. This single-step walk-through should be a standard step in QC, not something you do only when you suspect a problem.
2. Rolling features that roll the wrong direction
Rolling windows, lags, and moving averages are the workhorses of temporal features — and they’re exactly where direction gets confused. When you build a feature that summarizes history, you have to be certain you’re always looking backward, never forward.
The trap is how shifting works in pandas and numpy. A positive shift, a negative shift, min_periods, whether the window is centered — each of these can silently pull a future value into the current row. rolling(window).mean() followed by the wrong shift is a classic: your “trailing average” ends up including the current or next value.
The check: for any rolling or lagged feature, verify on a small example that the value at row t only draws from rows strictly before t (or up to and including t, if that’s genuinely available at inference — but be honest about which). Don’t trust the API defaults; confirm the direction by hand.
3. Confusing feature and target names
While experimenting you engineer a lot of columns, fast. In the churn, it is genuinely easy to feed a target back in as a feature — especially when a target is derived from the same raw signal as some of your features, or when a name is ambiguous enough that you forget which side it belongs on. The model then “predicts” something it was handed directly, and your metrics look unbelievable. They are.
The practice: name every feature with an x_ prefix and every target with a y_ prefix. Then selecting your feature matrix is df.filter(regex="^x_") and your targets are df.filter(regex="^y_") — there is no way to accidentally pull a y_ into the x_ set. (I’d love to take credit for this, but it was my colleagues’ idea. It has saved us more than once.)
4. Transformers: leakage baked into the attention
The seminal paper was titled Attention Is All You Need, and attention is precisely where leakage can hide when you move to transformer architectures. Self-attention lets every position in a sequence attend to every other position. That is the whole point — and for a forecasting or autoregressive model, it’s also the danger. Without constraints, the representation for time step t is computed with full access to t+1, t+2, … You have leaked the future into the present as a matter of architecture, before you’ve written a single feature.
The fixes are well established, but each is easy to get wrong:
- Causal (look-ahead) masking. In the decoder’s self-attention, apply a triangular mask so position t can only attend to positions ≤ t. Miss it, or apply it to the wrong axis, and every position sees ahead. This is the single most important guard, and it’s worth explicitly verifying the mask rather than trusting that the layer does it for you.
- Bidirectional encoders. BERT-style encoders attend in both directions by design. That’s fine for classifying a complete sequence, but if any representation that has seen the whole window is then used to predict a future point in that same window, you’ve leaked. Be deliberate about which parts of the sequence each representation was allowed to see.
- Normalization and scaling statistics. This one bites people constantly. If you standardize or normalize using mean/variance computed over the entire sequence — including future steps — the future has leaked in through the scaler, no matter how correct your attention mask is. Compute scaling statistics from the past only, or fit them on train and apply forward.
- Teacher forcing and target shifting. During training you feed the ground-truth sequence shifted by one. Off-by-one the shift and the model is trained to read the token it’s supposed to predict. Check the alignment on a tiny batch by hand.
The theme across all four: attention gives the model the ability to see everything, so with transformers, leakage is the default and correctness is something you have to actively impose and verify.
A few more places it hides
Once you start looking, leakage turns up in more corners than you’d expect:
- Global preprocessing before the split. Imputation, scaling, encoding, or feature selection fit on the full dataset before you carve out validation and test leaks statistics from the future into training.
- Target-derived features. Any feature computed from the same quantity you’re predicting — even a lag of it — needs the same backward-only scrutiny as everything else.
- Resampling and interpolation. Filling gaps or resampling to a coarser frequency can smear a future observation backward into a past timestamp.
- Leaky splits. Shuffling rows or doing a random split on time series puts near-future neighbors in the training set. Split by time, and leave a gap if your features have a lookback window that could straddle the boundary.
The one habit that catches most of it
If I had to keep a single practice, it would be the step-by-step walk-through from the very first point: pretend you are the model at inference, standing at time t with no knowledge of anything after it, and ask whether you could construct every feature you’re feeding in. Most leakage cannot survive that question honestly asked.
It’s a boring check. Run it anyway, every time — because the alternative is a model that looks brilliant in every report you write and falls apart the moment it meets the future for real.