The Confounding Problem: Why Your Experiments Can Fool You

The Experiment That Looked Convincing

A user ran a 45-day magnesium glycinate experiment. Took 400mg each evening on intervention days. Measured sleep score every morning from their Oura ring. The result: sleep scores averaged 73 on magnesium nights versus 68 on placebo nights. A 5-point difference. 82% posterior probability that magnesium helped. Looked convincing.

Then they noticed something. They'd started a new job three weeks into the experiment. And they'd been more likely to take their magnesium on days when work had gone well — less anxious, winding down earlier, not lying awake rehearsing tomorrow's tasks. The magnesium and the stress level had moved together. Not because of each other, but because the same underlying thing — a good versus difficult workday — caused both.

The magnesium may have done nothing. The stress level may have done everything. Or magnesium may have helped a little and the stress confounded the rest. With observational data entangled this way, there is no clean answer.

This is the confounding problem in personal experiments. It's not exotic. It happens constantly.

What Confounding Actually Is

A confounder is a variable that influences both which condition you end up in and what your outcome is.

The clearest way to see this: suppose you're testing whether your morning run improves your afternoon focus. You tend to run on days when you sleep well. You also tend to have better focus on days when you sleep well. Now you find that focus is higher on run days. Is it the running or the sleep? You can't tell from the data, because sleep quality is tangled with both the exposure (whether you ran) and the outcome (focus score).

Sleep quality here is a confounder. It's not the intervention. It's not the outcome. But it's driving a correlation between them that doesn't reflect a real causal relationship.

The same dynamic appears across almost every common personal experiment:

• Caffeine cutoff experiments: stress and workload affect both when you stop drinking coffee and how well you sleep
• Exercise timing experiments: motivation levels affect both when you work out and how you perform afterward
• Diet interventions: social context affects both what you eat and your mood
• Supplement experiments: how consistently you take a supplement is often correlated with how "on" you're feeling that day

If your condition assignment is not random — if you're choosing which condition to follow based on how you feel, or external conditions are pushing you toward one condition more than the other — confounders are almost certainly at work.

The Difference Between Confounding and Noise

Noise is random variation that makes your estimates imprecise. More data reduces noise. With enough trials, a real effect emerges through the noise.

Confounding is systematic bias. More data doesn't fix it — it just makes the biased estimate more precise. You get a very confident wrong answer.

This is why confounding is more dangerous than noise. A noisy experiment will produce uncertain results, which looks appropriately uncertain. A confounded experiment will produce confident results that point in the wrong direction. The statistical analysis has no way to know that the effect it's measuring isn't what you think it is.

The practical implication: running your experiment longer doesn't help if the confound is systematic. You need to address the source of the confounding, not collect more data around it.

The Classic Fix: Randomization

The reason randomized controlled trials are considered the gold standard is not that randomization is magical. It's that randomization breaks the link between confounders and condition assignment.

If you flip a coin each morning to decide whether today is a magnesium day or a placebo day, your stress level that day can't systematically push you toward one condition. On average, stressful days are equally distributed across the two conditions. The confounder becomes uncorrelated with the intervention. It adds noise but not bias.

This is why the randomization step in your experiment design matters more than most people realize. Convenience-based condition assignment — "I'll do the hard protocol when I feel ready for it" — is a recipe for confounded results. The coin flip feels unnecessary when you're logging every day. It is not.

When You Can't Randomize

Some interventions are hard to randomize. You can't flip a coin each morning and decide whether to practice intermittent fasting today. The protocol has multi-day structure. You have to commit to it for a run of consecutive days, then switch.

In these cases — crossover designs with block structure — the confounders tend to be trends over time: seasonal changes, work project cycles, relationship stress, training load building up. The fix is to measure those potential confounders and account for them.

The key variables to track alongside any experiment:

• Stress level (1–10, morning rating)
• Sleep quality the night before
• Alcohol the previous evening (binary or units)
• Exercise (yes/no, or intensity)
• Social plans (high-contact vs. low-contact day)
• Any work or life stressor that's variable across the experiment period

You don't need to track all of these. You need to track the ones that are likely to co-vary with your condition assignment. If you run more on weekends and you're testing a protocol that also happens to change on weekends, week-type is a confounder you should log.

Reading the Warning Signs

A few patterns in your data should raise confound suspicion:

Your condition assignment is predictable from your context. If you can predict, before looking at outcomes, that intervention days cluster on weekends / low-stress periods / times when you're sleeping well — your condition isn't effectively randomized. Whatever drives that clustering is a likely confounder.

Your outcome metric is highly correlated with a tracked confounder. If your focus score tracks your stress level at r = 0.6, and your stress level also tracks whether you did your focus protocol, the protocol's apparent effect is partially or fully explained by stress.

The effect varies dramatically depending on who was in a given condition. If your "caffeine-free days" happen to be your low-workload days and vice versa, the caffeine effect estimate is really a workload effect in disguise.

The effect disappears when you control for a confounder. If including stress level as a covariate in your analysis reduces the apparent intervention effect from 1.2 to 0.3 points, stress was doing most of the work.

What to Do With a Potentially Confounded Result

If you've already run an experiment and you're suspicious that confounding is distorting the results:

Step 1: Check your data for the confound. Look at whether the confounder value is systematically different across your two conditions. Plot stress-by-condition, sleep-by-condition, whatever seems likely. If the distributions are similar, confounding from that variable is probably minimal. If they're clearly different, you have a problem.

Step 2: Adjust statistically if you have enough data. With 40+ trials per condition and a tracked confounder, you can partial out the confounder's effect using regression adjustment. This gives you an estimate of the intervention effect holding the confounder constant. It's not as clean as having randomized in the first place, but it's far better than ignoring the problem.

Step 3: Re-run with better controls. For important questions, a re-run with strict randomization and explicit confounder tracking will give you a cleaner answer than any amount of post-hoc adjustment on the original data.

The goal is not a perfect experiment. The goal is an experiment whose results you can interpret honestly. Knowing where your confounders are is half the battle — an identified confounder is a problem you can manage. An unidentified one is a silent distortion in your conclusions.

---

Steady Practice now automatically flags confounders that are correlated with your condition assignment (r ≥ 0.3). You'll see these as warnings in your experiment analysis — a starting point for deciding whether a re-run or adjustment is warranted. Start a new experiment or check your existing results.