Asthma as a Protective Factor
In the 1990s, a group of researchers trained several models to predict which pneumonia patients were likely to die, so that low-risk patients could be treated at home. Rich Caruana, one of them, described what went wrong in a KDD 2015 paper. A rule-based model learned that patients with asthma had a lower risk of death. The rule was correct about the data. Asthmatic patients with pneumonia were usually sent straight to intensive care, and the aggressive treatment lowered their mortality below the average.
A model that sent asthmatic patients home because of that rule would have removed the very treatment that produced the low risk. The rule-based model was readable, so a person caught it. The most accurate model in the study, a neural network, had probably learned the same pattern where nobody could see it.
The model answered the question it was trained on: among patients like this, how often do they die? The hospital needed an answer to a different question: what happens to this patient if we treat them one way or the other? The first is prediction. The second is causal inference, and no amount of extra data of the first kind settles it.
Case: Kidney Stones
A 1986 study in the BMJ by C. R. Charig and colleagues compared two treatments for kidney stones. Open surgery succeeded in 78% of 350 patients. Percutaneous nephrolithotomy, a less invasive procedure, succeeded in 83% of 350 patients. The newer method looks better.
Split the patients by stone size and the comparison flips. For small stones, open surgery succeeded in 81 of 87 cases (93%) against 234 of 270 (87%) for the other procedure. For large stones, it was 192 of 263 (73%) against 55 of 80 (69%). Open surgery wins in both groups and loses overall.
This is Simpson’s paradox, and the explanation is in how patients were assigned. Doctors tended to use open surgery on large stones, which are harder to treat no matter what. Stone size affected both the choice of treatment and the outcome. A table of numbers cannot say whether to trust the overall figure or the split one. That depends on what causes what, and the data do not contain that information. A person has to bring it.
Potential Outcomes
One formal language for this comes from statistics. Jerzy Neyman used it for agricultural experiments in 1923, and Donald Rubin developed it for observational studies in a 1974 paper in the Journal of Educational Psychology. Each patient has two potential outcomes: the result they would have with the treatment and the result they would have without it. The causal effect for that patient is the difference.
The difficulty is that only one of the two is ever observed. A patient who had open surgery did not also have the other procedure. Paul Holland, writing in 1986, called this the fundamental problem of causal inference. Every causal method is a way of filling in the missing half from other people who received the other treatment.
A randomized experiment fills it in cleanly, because a coin flip decides who gets what, so the two groups are alike on average in everything, measured or not. Without randomization, the usual assumption is called unconfoundedness: once a set of measured variables is accounted for, treatment assignment is as good as random. It cannot be tested from the data. It is a claim about the world.
Graphs and the do Operator
Judea Pearl approached the same problem from computer science. He represented assumptions as a directed graph in which an arrow from A to B means A directly influences B. In the kidney stone study, stone size points to treatment and to outcome, and treatment points to outcome.
Pearl separated two quantities that ordinary probability writes the same way. The probability of recovery given that a patient received a treatment describes the patients doctors chose for it. The probability of recovery given do(treatment) describes what would happen if everyone were assigned to it. His back-door criterion says which variables to adjust for to turn the first into the second. Here, adjusting for stone size is correct, so the split table gives the right answer. In a different graph, where the third variable is caused by the treatment, adjusting for it would create bias instead of removing it.
In a 1995 paper in Biometrika, Pearl introduced the do-calculus, three rules for deciding whether an interventional quantity can be computed from observational data at all. In his 2018 book with Dana Mackenzie, The Book of Why, he arranges these questions on a ladder with three rungs: association (seeing), intervention (doing), and counterfactuals (imagining what would have happened). Standard supervised learning stays on the first rung.
Case: LaLonde and the Job Training Program
The most cited test of whether observational methods work comes from economics. The National Supported Work Demonstration was a job training program in the 1970s that randomly assigned applicants to training or to a control group, so the true effect on earnings was known. In a 1986 paper in the American Economic Review, Robert LaLonde threw away the randomized control group, replaced it with comparison people taken from large national surveys, and ran the standard econometric estimators of the time. Many of them failed to recover the experimental result, and different specifications gave very different answers.
Paul Rosenbaum and Donald Rubin had proposed the propensity score in 1983: the probability that a person receives the treatment given their covariates. Rajeev Dehejia and Sadek Wahba applied it to LaLonde’s data in a 1999 JASA paper and got estimates much closer to the experimental benchmark.
The story did not end there. Jeffrey Smith and Petra Todd argued in 2005 that the good result depended on which subsample of LaLonde’s data was used. The lesson students take from the debate is narrower than «matching works.» The estimator matters less than whether the comparison group resembles the treated group in the ways that drive the outcome.
Machine Learning as the Estimator
Once the causal question and its assumptions are fixed, what is left is estimation, and that is where machine learning fits well. Adjusting for confounders requires modeling how the outcome depends on them and how treatment depends on them.
Simply plugging a flexible model into the old formulas causes trouble, because regularization biases the estimate in ways that do not average out. Victor Chernozhukov and six coauthors addressed this in «Double/Debiased Machine Learning,» published in The Econometrics Journal in 2018. They fit two machine learning models, one for the outcome and one for the treatment, and estimate the effect from the residuals of both. They also use cross-fitting, where the models are trained on one part of the data and evaluated on another. The result is an estimate with valid confidence intervals even when the models are random forests or neural networks.
A second line of work asks whether the effect differs from person to person. Stefan Wager and Susan Athey extended random forests into causal forests (JASA, 2018). The trees split the data to find groups where the treatment effect differs, and the authors proved the estimates are consistent and approximately normal under unconfoundedness. Marketing teams use the same idea under the name uplift modeling, where the goal is to find customers whose behavior changes because of a discount, not customers who would have bought anyway.
Shortcuts in Image Models
Deep learning has its own version of the asthma rule. John Zech and colleagues (PLOS Medicine, 2018) trained convolutional networks to detect pneumonia in chest X-rays from three hospital systems. Performance on X-rays from a hospital not seen in training was significantly worse in three of five comparisons. The networks could also identify which hospital an image came from with over 99.9% accuracy. Since pneumonia rates differed between hospitals, recognizing the hospital was a cheap way to predict the disease.
In causal terms, the hospital is a confounder that affects both the image and the label. The network learned a correlation that held in the training data and broke when the data came from somewhere else.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani and David Lopez-Paz proposed Invariant Risk Minimization in 2019 as one response. If data come from several environments, a feature that predicts the label equally well in all of them is more likely to be causal than one whose usefulness changes. Their test case, Colored MNIST, ties the color of a digit to its label in training and reverses the link at test time. Later analyses found that the practical version of IRM often does no better than ordinary training, so the problem is far from solved.
Learning the Graph From Data
All of the methods above assume someone drew the graph. Causal discovery tries to recover it from data. The PC algorithm, developed by Peter Spirtes and Clark Glymour around 1991, tests conditional independences and removes edges between variables that turn out to be independent given others. In 2018, Xun Zheng and colleagues at NeurIPS reformulated the search over graphs as a continuous optimization problem, called NOTEARS, which can be solved with gradient methods.
There is a hard limit. Observational data alone usually cannot distinguish between graphs that imply the same independences. A chain from A to B to C and a chain from C to B to A produce the same pattern, so the algorithm can return a set of equivalent graphs but not choose among them. Breaking the tie takes an experiment or an extra assumption.
A Final Observation
Every causal estimate in this post depends on something outside the data: a graph, an assumption of unconfoundedness, a randomized control group someone paid for. Machine learning has made the estimation step much more powerful, and it has not changed that. The kidney stone table is the same table whether one trusts the overall figure or the split one. What decides between them is a sentence a doctor can say and a dataset cannot: large stones were harder to treat and more often sent to surgery. Causal inference is the part of machine learning where the most important input is written down by a person.
By: Max Johnson B.
Deja una respuesta