The Pitfalls of Observational Studies – Watts Up With That?

Age, sex, and self-reported race/ethnicity were identified from health plan databases. Race/ethnicity was included because studies suggest it may be associated with differential treatment or outcomes in cardiovascular diseases. Socioeconomic status was estimated from 2000 US Census data. Low education was defined as living in a census block where more than 25% of those aged 25 years or older had less than a 12th-grade education; low income was defined as living in a block where annual household income is less than $35 000 per year.²² We ascertained information on coexisting illnesses based on diagnoses or procedures using ICD-9 codes, laboratory results, or specific therapies from health plan hospitalization discharge, ambulatory visit, laboratory, and pharmacy databases; diabetes mellitus registry²³; and regional cancer registry.²⁴ This included baseline and follow-up diagnoses of CHD, cerebrovascular disease, peripheral arterial disease, diabetes, hypertension, malignancy, thyroid disease, liver disease, lung disease, human immunodeficiency virus infection, valvular disease, dementia, depression, ventricular arrhythmias, and atrial fibrillation/flutter (ICD-9 codes available on request).

We also identified outpatient measurements of total cholesterol, low-density lipoprotein cholesterol, high-density lipoprotein cholesterol, and hemoglobin¹⁷ from laboratory databases during the 12 months before study entry and throughout follow-up. We classified kidney function using the Modification of Diet in Renal Disease equation for estimated glomerular filtration rate based on outpatient determinations of serum creatinine.^25,26 …

We ascertained information on systolic function status from health plan databases. If available, we used a previously described approach¹⁸ to define reduced systolic function as left ventricular ejection fraction of less than 40% or a qualitative description of moderate or severely reduced systolic function; preserved left ventricular systolic function was defined by left ventricular ejection fraction of 40% or higher or a qualitative description of normal or only mildly reduced systolic function.

Finally, as a proxy for intensity of care, we identified the number of visits to a cardiologist before study entry and during follow-up. Information on participation in heart failure case management programs was not available.

[46] Book: Introductory Econometrics: Using Monte Carlo Simulation with Microsoft Excel. By Humberto Barreto and Frank M. Howland. Cambridge University Press, 2006.

Page 491:

Omitted variable bias is a crucial topic because almost every study in econometrics is an observational study as opposed to a controlled experiment. Very often, economists would like to be able to interpret the comparisons they make as if they were the outcomes of controlled experiments. In a properly conducted controlled experiment, the only systematic difference between groups results from the treatment under investigation; all other variation stems from chance. In an observational study, because the participants self-select into groups, it is always possible that varying average outcomes between groups result from systematic difference between groups other than the treatment. We can attempt to control for these systematic differences by explicitly incorporating variables in a regression. Unfortunately, if not all of those differences have been controlled for in the analysis, we are vulnerable to the devastating effects of omitted variable bias.

[47] Book: Multiple Regression: A Primer. By Paul D. Allison. Pine Forge Press, 1998.

Chapter 1: “What Is Multiple Regression?” https://us.sagepub.com/sites/default/files/upm-binaries/2725_allis01.pdf

Page 20:

Multiple regression shares an additional problem with all methods of statistical control, a problem that is the major focus of those who claim that multiple regression will never be a good substitute for the randomized experiment. To statistically control for a variable, you have to be able to measure that variable so that you can explicitly build it into the data analysis, either by putting it in the regression equation or by using it to form homogeneous subgroups. Unfortunately, there’s no way that we can measure all the variables that might conceivably affect the dependent variable. No matter how many variables we include in a regression equation, someone can always come along and say, “Yes, but you neglected to control for variable X and I feel certain that your results would have been different if you had done so.”

That’s not the case with randomization in an experimental setting. Randomization controls for all characteristics of the experimental subjects, regardless of whether those characteristics can be measured. Thus, with randomization there’s no need to worry about whether those in the treatment group are smarter, more popular, more achievement oriented, or more alienated than those in the control group (assuming, of course, that there are enough subjects in the experiment to allow randomization to do its job effectively).

[48] Book: Theory-Based Data Analysis for the Social Sciences (2nd edition). By Carol S. Aneshensel. SAGE Publication, 2013.

Page 90:

The numerous variables that are omitted from any model are routinely assumed to be uncorrelated with the error term, a requirement for obtaining unbiased parameter estimates from regression models. However, the possibility that unmeasured variables are correlated with variables that are in the model obviously cannot be eliminated on empirical grounds. Thus, omitted variable bias cannot be ruled out entirely as a counterargument for the empirical association between the focal independent and dependent variables in observational studies.

[49] Book: Applied Statistics for Economists. By Margaret Lewis. Routledge, 2012.

In economics, our primary concern is to identify and then include all relevant independent variables as indicated by economic theory.⁹ Omitting such variables will cause the regression model to be underspecified, with the partial regression coefficients that are affected by the omitted variable(s) will not equal the true population parameters.

[50] Encyclopedia of Education Economics and Finance. Edited by Dominic J. Brewer and Lawrence O. Picus. Sage Publications, 2014.

Page 498:

Omitted variable bias (OVB) occurs when an important independent variable is excluded from an estimation model, such as a linear regression, and its exclusion causes the estimated effects of the included independent variables to be biased. Bias will occur when the excluded variable is correlated with one or more of the included variables. An example of this occurs when investigating the returns to education. This typically involves regressing the log of wages on the number of years of completed schooling as well as on other demographic characteristics such as an individual’s race and gender. One important variable determining wages, however, is a person’s ability. In many such regressions, a measure of ability is not included in the regression (or the measure included only imperfectly controls for ability). Since ability is also likely to be correlated with the amount of schooling an individual receives, the estimated return to years of completed schooling will likely suffer from OVB.

[51] Book: Higher Education: Handbook of Theory and Research (Volume 28). Edited by Michael B. Paulsen. Springer, 2013.

Chapter 6: “Instrumental Variables: Conceptual Issues and an Application Considering High School Course Taking.” By Rob M. Bielby and others. Pages 263–312.

Page 273:

An additional issue with the aforementioned studies is that none employ strategies to eliminate the influence of unobservable factors on course taking and attainment. Some student characteristics may be difficult or impossible to obtain information about in observational datasets, but this does not change the fact that they are confounding factors (Cellini, 2008). Examples of potential unobservable factors in course taking effects research include a student’s enjoyment of the learning process and a student’s desire to undertake and persevere through challenges. It is likely that these unobservable factors contribute to student selection into high school courses and a student’s subsequent choice to attain a bachelor’s degree. However, none of the studies we examined that employ a standard regression approach accounted for a student’s intrinsic love of learning or ability to endure through difficulties; the failure to account for these unobserved factors may bias the estimates that result from these studies.

[52] Paper: “Observational Research Rigour Alone Does Not Justify Causal Inference.” By Keisuke Ejima and others. European Journal of Clinical Investigation, October 6, 2016. https://onlinelibrary.wiley.com/doi/10.1111/eci.12681

The greatest challenge to drawing causal inferences in observational studies is the existence of potential confounding variables, not all of which can be specified, measured, or modeled. Randomization is the only method that can eliminate all potential confounders of the effect of treatment assignment per se, doing so by making the distribution of prerandomization factors identical for all treatment assignments at the population level [2].

[53] Commentary: “Observational Studies, Bad Science, and the Media.” By Steven E. Nissen, MD. American College of Cardiology, May 25, 2012. https://www.acc.org/Latest-in-Cardiology/Articles/2012/05/25/12/44/Observational-Studies

The limitations of observational studies are myriad, but the most common flaws are easily understood and explained. Since patients are not randomly assigned to a treatment group, there always exist differences in characteristics between the study groups. The best observational studies attempt to adjust for these “confounders,” but often consider only the most common demographic variables, such as age and gender. Statistical adjustment can never fully compensate for all of the differences in patient characteristics, leading a common problem known as “residual confounding.”

[54] Paper: “The Phantom Menace: Omitted Variable Bias in Econometric Research.” By Kevin A. Clarke. Conflict Management and Peace Science, September 2005. https://journals.sagepub.com/doi/10.1080/07388940500339183

Quantitative political science is awash in control variables. The justification for these bloated specifications is usually the fear of omitted variable bias. A key underlying assumption is that the danger posed by omitted variable bias can be ameliorated by the inclusion of relevant control variables. Unfortunately, as this article demonstrates, there is nothing in the mathematics of regression analysis that supports this conclusion. The inclusion of additional control variables may increase or decrease the bias, and we cannot know for sure which is the case in any particular situation.

[55] Paper: “Observational Research Rigour Alone Does Not Justify Causal Inference.” By Keisuke Ejima and others. European Journal of Clinical Investigation, October 6, 2016. https://onlinelibrary.wiley.com/doi/10.1111/eci.12681

Differing opinions exist on whether associations obtained in observational studies can be reliable indicators of a causal effect if the observational study is sufficiently well controlled and executed. …

To test this, we conducted two animal observational studies that were rigorously controlled and executed beyond what is achieved in studies of humans. …

In both studies we observed markedly disparate results of the observational association estimates and the experimental effect estimates regarding food consumption effects on lifespan and weight gain. With the randomized experimental design as the gold standard of causal inference, the observational association estimates in study 1 showed the opposite direction of the “true effect” of daily energy intake on lifespan in mice, whereas those estimates in study 2 did not represent the “true effect” of assigned food consumption on weight gain in young mice. Even [if] we employed a study design using genetically identical mice in the same environment; it is very possible that there are some unmeasured confounders not controlled in our observational designs, such as undiagnosed disease or individual characteristics, which can be strongly correlated to both exposure and outcomes and eventually bias the statistical inferences. For example, mice with any undiagnosed diseases or specific metabolic characteristics (e.g., basal metabolic rate, daily energy expenditure) may eat poorly and die earlier than the other healthy mice, which would produce a biased result. These two studies are typical examples of un-controlled confounding in studies with self-selection feature, which is frequently involved in food consumption research. The un-controlled confounders in self-selected food consumption (or other factors more generally) apparently can result in biased inference. Thus, relying on self-selection can lead to biased estimation of causal effects. …

Observational designs remain useful in biomedical and behavioral research by allowing empirical investigations of exposures and their associations in populations experiencing all the vagaries of everyday life. Observational studies offer potential directions for randomized controlled experimental studies. However, as we have illustrated here, even an observational study that is meticulously controlled far beyond what could be achieved in a human study cannot be counted upon to reliably estimate causal effects owing to uncontrolled confounders, especially in nutrition research. Therefore, we believe that, despite public statements to the contrary [6], observational studies alone, no matter how well done, cannot support conclusions of causation.

[56] Book: The Education Gap: Vouchers and Urban Schools (Revised edition). By William G. Howell and Paul E. Peterson with Patrick J. Wolf and David E. Campbell. Brookings Institution Press, 2006 (first published in 2002). https://www.brookings.edu/book/the-education-gap/

Page 39:

In a perfectly controlled experiment in the natural sciences, the researcher is able to control for all factors while manipulating the variable of interest. …

Experiments with humans are much more difficult to manage. Researchers cannot give out pills or placebos and then ask subjects not to change any other aspect of their lives.

[57] Textbook: Criminology. By Steve Case and others. Oxford University Press, 2017.

Chapter 27: “Searching for the Causes of Crime.” Pages 525–548.

Page 534: “The statistical relationships identified between different factors and offending behaviors/crime are correlational, not causal.”

Page 546: “To answer the title question, searching for the cause of crime is an impossible goal in the strict experimental, positivist sense, because there are simply too many unknowns and unmeasured dark figures of crime and explanation to enable us to draw valid and reliable conclusions from research.”

[58] Paper: “Resisting Rape: The Effects of Victim Self-Protection on Rape Completion and Injury.” By Jongyeon Tark and Gary Kleck. Violence Against Women, March 2014. https://journals.sagepub.com/doi/epdf/10.1177/1077801214526050

Page 270:

The impact of victim resistance on rape completion and injury was examined utilizing a large probability sample of sexual assault incidents, derived from the National Crime Victimization Survey … and taking into account whether harm to the victim followed or preceded self-protection (SP) actions. Additional injuries besides rape, particularly serious injuries, following victim resistance are rare. Results indicate that most SP actions, both forceful and nonforceful, reduce the risk of rape completion, and do not significantly affect the risk of additional injury. …

Finally, given the impossibility of experimental research on this topic, it should be noted that our findings are necessarily based on observed associations between victim actions and assault outcomes, thereby precluding definitive conclusions about causal effects.

[59] Paper: “Association Is Not Causation: Treatment Effects Cannot Be Estimated From Observational Data in Heart Failure.” By Christopher J Rush and others. European Heart Journal, October 2018. https://academic.oup.com/eurheartj/article/39/37/3417/5063542

Treatment ‘effects’ are often inferred from non-randomized and observational studies. These studies have inherent biases and limitations, which may make therapeutic inferences based on their results unreliable. We compared the conflicting findings of these studies to those of prospective randomized controlled trials (RCTs) in relation to pharmacological treatments for heart failure (HF). …

We searched Medline and Embase to identify studies of the association between non-randomized drug therapy and all-cause mortality in patients with HF until 31 December 2017. … We identified 92 publications, reporting 94 non-randomized studies, describing 158 estimates of the ‘effect’ of the six treatments of interest on all-cause mortality, i.e. some studies examined more than one treatment and/or HF phenotype. These six treatments had been tested in 25 RCTs. For example, two pivotal RCTs showed that MRAs reduced mortality in patients with HF with reduced ejection fraction. However, only one of 12 non-randomized studies found that MRAs were of benefit, with 10 finding a neutral effect, and one a harmful effect.

This comprehensive comparison of studies of non-randomized data with the findings of RCTs in HF shows that it is not possible to make reliable therapeutic inferences from observational associations. While trials undoubtedly leave gaps in evidence and enroll selected participants, they clearly remain the best guide to the treatment of patients.

[60] Paper: “The Phantom Menace: Omitted Variable Bias in Econometric Research.” By Kevin A. Clarke. Conflict Management and Peace Science, September 2005. https://journals.sagepub.com/doi/10.1080/07388940500339183

Quantitative political science is awash in control variables. The justification for these bloated specifications is usually the fear of omitted variable bias. A key underlying assumption is that the danger posed by omitted variable bias can be ameliorated by the inclusion of relevant control variables. Unfortunately, as this article demonstrates, there is nothing in the mathematics of regression analysis that supports this conclusion. The inclusion of additional control variables may increase or decrease the bias, and we cannot know for sure which is the case in any particular situation.

[61] Book: Multiple Regression: A Primer. By Paul D. Allison. Pine Forge Press, 1998.

Chapter 3: “What Can Go Wrong With Multiple Regression?” https://us.sagepub.com/sites/default/files/upm-binaries/2726_allis03.pdf

Pages 60–61:

Do Some Variables Mediate the Effects of Other Variables?

Even if the sample is not small, there is another reason for being cautious in concluding that a variable has no effect: It’s possible that other variables mediate the effect of that variable. If those other variables are also included in the regression model, the effect of the variable you’re interested in may disappear.

[62] Paper: “Understanding How the Social Scientific Study of Same-Sex Parenting Works.” By Mark Regnerus. Annals of Social Sciences / Roczniki Nauk Społecznych, 2020. https://ojs.tnkul.pl/index.php/rns/article/download/15976/15377/

[T]he “consensus” that children from same-sex households fare no differently than children from opposite-sex households—in particular, married families—is a carefully guarded social construction. The consensus is the result of sampling decisions, analytic comparisons, and interpretations of results that often indicate baseline differences prior to statistical controls for household instability, after which they commonly disappear. …

The story of “no differences” between same-sex and opposite-sex households with children hinges on a pair of repetitive themes in the published research: small and nonrepresentative sampling strategies, and analytic strategies that all but guarantee the ability to “explain away” any baseline observable differences between children from same-sex and opposite-sex households. …

Where deficits would be considered consequential and potentially harmful to admit, controlling for household instability is nearly ubiquitous. For instance, a 2016 study using data from the population-based Early Childhood Longitudinal Study—Kindergarten cohort found that the children of same-sex parents wellbeing measured significantly lower on measures such as interpersonal skills and both internalizing and externalizing wellbeing when compared to the children of married opposite-sex parents. But then the authors employ the familiar tactic:

After including family change and early childhood transitions in the model, differences in the externalizing well-being, internalizing well-being, and interpersonal skills of children in same-sex parent households were no longer significantly different from their peers in married, two-biological parent families.³¹

This is how researchers get to no “differences,” that is, by controlling for—hence ignoring—household turmoil.

Others simply obscure the results in their discussion of them. For example, in a 2018 study which compared the mental health of the NLLFS’s [National Longitudinal Lesbian Family Study’s] then-25-year-old donor-conceived children of lesbian parents to a population-based sample of children from opposite-sex households, the authors reported “no significant differences in measures of mental health” between the two groups.³² And yet the evidence presented in the study itself reveals that the NLLFS children reported demonstrably higher levels of “depression or anxiety” than the control group.³³ Nevertheless, the next year the same authors mentioned how their 2018 study conclusions “provide no justification for restricted access to reproductive technologies, adoption, foster care, or civil liberties for lesbian, gay, or bisexual people.”³⁴

My concern is not at all with the use of control variables and regression analyses. These are, after all, standard approaches. The problem is that this method is often misemployed to “control away” how reality and social processes work, and to ensure a high likelihood of a “no differences” conclusion.

The most common way this occurs is by controlling for parental relationship dissolution, rates of which vary dramatically between gay and straight couples. How dramatically? Estimates vary, but they never reveal lower breakup rates among same-sex households with children. A 2020 study of over 1.2 million children in the (gay-friendly) Netherlands revealed that 55 percent of children living with same-sex parents—the vast majority of which were female couples—experienced parental separation, well above the 19 percent of children of opposite-sex parents who experienced the same.³⁵ The story in the Netherlands has not changed; the same pattern was observed using data from no later than 2000 in which (mostly cohabiting) same-sex couples experienced 3.1 times higher dissolution odds than opposite-sex cohabiting couples and 11.5 times higher odds compared with married couples.³⁶

One may claim, following minority stress theory, that if societies were more tolerant, sexual minorities wouldn’t feel the need to hide their identities and enter heterosexual relationships, only to see them “inevitably” fail, followed by the formation of relationships considered more “authentic.” But this is the Netherlands—it doesn’t get more tolerant than that. Even data from Sweden shows that women in same-sex marriages have a divorce rate nearly twice that of opposite-sex married couples.³⁷ And despite the fact that the NLLFS drew upon a particularly privileged set of recruited American lesbians, 62 percent of the young adults in that study reported in 2019 that their parents—typically a biological and social mother—had already broken up, a rate well above what we would expect to see among the offspring of opposite-sex parents.³⁸

A recent re-examination of three nationally-representative datasets from the United States and Canada similarly revealed that dissolution rates were different—but not profoundly so—for couples with no children: 9% for same-sex vs. 5% for opposite sex in one study, 27% and 17% in another, respectively. However, for couples with children (in a formalized union), the results were strikingly different, with dissolution rates of 43% for same-sex couples vs. 8% for opposite-sex. The presence of children tended to stabilize opposite-sex couples, but destabilize same-sex couples.³⁹ The authors suggest that “parental instability is an important factor through which parents’ sexual orientation influences children’s outcomes.”⁴⁰

The elevated break-up rate of female same-sex couples is a central mechanism here that is under-theorized. The consistent story is not about a direct effect of sexual orientation on children’s outcomes, but rather about the indirect effects of consolidating sex (or gender) preferences and behaviors. Even key proponents of the “no differences consensus” had long predicted this pattern, stating their suspicion that the “asymmetrical biological and legal statuses” and “high standards of equality” present in lesbian relationships would put them at a heightened risk of dissolution.⁴¹ Their suspicion has proven to be correct.

It is plausible, even likely, that household instability—via parental romantic- relationship fragility—is a key pathway or mechanism by which children come to have difficulties in one or more domains of life. This tendency to overlook pathways in favor of controls reflects a typical misguided tendency in social science research to always search for “independent” effects of variables, thereby missing the ways in which social phenomena actually operate and outcomes come to be.⁴² Controlling for the effect of a parent’s same-sex relationship with a “household instability” variable and concluding that there are “no differences” between children of same-sex and opposite-sex parents is tantamount to “controlling for the pathways.” It is unhelpful for describing and understanding how social reality works.⁴³

[63] As supposed evidence that Mexican immigrants are “less likely to commit crimes than the native-born,” a 2016 article by PolitiFact cited an American Immigration Council report and wrote that:

2010 Census data that shows incarceration rates of young, less educated Mexican, Salvadoran and Guatemalan men—which comprise the bulk of the unauthorized population—are “significantly lower” than incarceration rates of native-born young men without a high-school diploma.

If the claim above is accurate, it would not show that Mexican immigrants are “less likely to commit crimes than the native-born.” It would show that young, male Mexican immigrants with low education who remain in the U.S. are less likely to be incarcerated than native-born young men without a high-school diploma. With regard to:

• education:
  • In 2012, 54% of Mexican and Central American immigrants aged 25–64 did not have a high school diploma or equivalent, as compared to 7% of people born in the U.S. in the same age group.
  • In 2009, 65% of male prison inmates did not have a high school diploma, as compared to 19% of adults in the general population.
• remaining in the U.S.:

• In the decade from 2006 to 2015, the federal government deported 1,504,934 non-citizens who were convicted of committing crimes in the U.S. This was 10 times the number of non-citizens in U.S. adult correctional facilities at the end of this period (151,324).
• After convicts are released from state prisons, roughly 81% of them are arrested within a decade for committing completely new crimes (not probation or parole violations), with an average of five arrests per released prisoner.

[64] In October 2022, the British Journal of Sports Medicine published a small study which found that transgender women (i.e., males) on long-term estrogen therapy had higher cardiopulmonary capacity and grip strength than biological women.

In the words of the study’s authors, “These findings add new insights to the sparse information available on a highly controversial topic about the participation of TW [transgender women] in physical activities.”

Several months later in February 2023, the journal published a “correction” which stated that there was “no difference” in cardiopulmonary capacity or grip strength between the transgender women and biological women when “adjusted for fat-free mass.”

Controlling for fat-free mass obscures the fact that women generally have much higher body fat percentages than men. This gives biological men a competitive advantage in physical activities.

[65] Book: Multiple Regression: A Primer. By Paul D. Allison. Pine Forge Press, 1998.

Chapter 1: “What Is Multiple Regression?” https://us.sagepub.com/sites/default/files/upm-binaries/2725_allis01.pdf

Page 1: “Multiple regression is a statistical method for studying the relationship between a single dependent variable and one or more independent variables. It is unquestionably the most widely used statistical technique in the social sciences. It is also widely used in the biological and physical sciences.”

Chapter 3: “What Can Go Wrong With Multiple Regression?” https://us.sagepub.com/sites/default/files/upm-binaries/2726_allis03.pdf

Page 49:

Any tool as widely used as multiple regression is bound to be frequently misused. Nowadays, statistical packages are so user-friendly that anyone can perform a multiple regression with a few mouse clicks. As a result, many researchers apply multiple regression to their data with little understanding of the underlying assumptions or the possible pitfalls. Although the review process for scientific journals is supposed to weed out papers with incorrect or misleading statistical methods, it often happens that the referees themselves have insufficient statistical expertise or are simply too rushed to catch the more subtle errors. The upshot is that you need to cast a critical eye on the results of any multiple regression, especially those you run yourself.

Fortunately, the questions that you need to ask are neither extremely technical nor large in number. They do require careful thought, however, which explains why even experts occasionally make mistakes or overlook the obvious. Virtually all the questions have to do with situations where multiple regression is used to make causal inferences.

NOTE: Pages 49–65 detail eight possible pitfalls of regression analyses.

Page 65: “The preceding eight problems are the ones I believe most often lead to serious errors in judging the results of a multiple regression. By no means do they exhaust the possible pitfalls that may arise. Before concluding this chapter, I’ll briefly mention a few others.”

[66] Paper: “Econometric Methods for Causal Evaluation of Education Policies and Practices: A Non-Technical Guide.” By Martin Schlotter, Guido Schwerdt, and Ludger Woessmann. Education Economics, January 2011. https://www.tandfonline.com/doi/10.1080/09645292.2010.511821

Page 111:

Whenever other reasons exist that give rise to some correlation between the two things of interest—the treatment and the outcome—the overall correlation cannot be interpreted as the causal effect of the treatment on the outcome. Broadly speaking, this is what economists call the ‘endogeneity problem’. The term stems from the idea that treatment cannot be viewed as exogenous to the model of interest, as it should be, but that it is rather endogenously determined within the model—depending on the outcome or being jointly determined with the outcome by a third factor. Because of the problem of endogeneity, estimates of the association between treatment and outcome based on correlations will be biased estimates of the causal effect of treatment on outcome.²

Standard approaches try to deal with this problem by observing the other sources of possible correlation and take out the difference in outcomes that can be attributed to these other observed differences. This is the approach of multivariate models that estimate the effects of multiple variables on the outcome at the same time, such as the classical ordinary least-squares (OLS) or multilevel modeling (or hierarchical linear models, HLM) techniques. They allow estimating the association between treatment and outcome conditional on the effects of the other observed factors.

² Other possible sources of endogeneity include self-selection (objects with different characteristics can choose whether to be treated or not) and simultaneity (treatment and outcome are choice variables that are jointly determined). In econometric terms, measurement error in the treatment variable can also be interpreted as an endogeneity problem, because it gives rise to a particular form of association between treatment and outcome (one that generally biases the estimates toward finding no effect, even if there was one).

[67] Article: “Statistical Malpractice.” By Bruce G. Charlton. Journal of the Royal College of Physicians of London, March 1996. https://www.researchgate.net/publication/14493406_Statistical_malpractice

Page 112: “Science is concerned with causes but statistics is concerned with correlations.”

Page 113:

The root of most instances of statistical malpractice is the breaking of mathematical neutrality and the introduction of causal assumptions into analysis without justifying them on scientific grounds. This amounts to performing science by sleight-of-hand: the quickness of the statistics deceives the mind. The process is often accidental, the product of misunderstanding rather than of malice—as commonly happens when statistical adjustments or standardization of populations are performed to remove the effects of confounding variables.⁶ These are maneuvers by which data sets are recalculated (for example, by stratified or multivariate analysis) in an attempt to eliminate the consequences of uncontrolled “interfering” variables which distort the causal relationship under study. …

There are however, no statistical rules by which confounders can be identified, and the process of adjustment involves making quantitative causal assumptions based upon secondary analysis of the database in question. …

Adjustment is therefore, implicitly, a way of modeling the magnitude of a causal process in order to subtract its effects from the data. However, modeling is not mathematically neutral and involves inputting assumptions—an activity which requires to be justified for each case. …

… Statistical malpractice occurs, however, exactly because it has not dispensed with causation, but has merely concealed it under a cloak of mathematical neutrality.

[68] Paper: “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” By Raphael R. Silberzahn and others. Advances in Methods and Practices in Psychological Science, August 23, 2018. https://journals.sagepub.com/doi/full/10.1177/2515245917747646

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. … Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. … These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. …

… The analytic techniques chosen ranged from simple linear regression to complex multilevel regression and Bayesian approaches. The teams also varied greatly in their decisions regarding which covariates to include….

Figure 2 shows each team’s estimated effect size, along with its 95% confidence interval (CI). As this figure and Table 3 show, the estimated effect sizes ranged from 0.89 (slightly negative) to 2.93 (moderately positive) in odds-ratio (OR) units; the median estimate was 1.31. The confidence intervals for many of the estimates overlap, which is expected because they are based on the same data. Twenty teams (69%) found a significant positive relationship, p < .05, and nine teams (31%) found a nonsignificant relationship. No team reported a significant negative relationship. …

Table 3. Analytic Approaches and Results for Each Team

[69] Paper: “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” By Raphael R. Silberzahn and others. Advances in Methods and Practices in Psychological Science, August 23, 2018. https://journals.sagepub.com/doi/full/10.1177/2515245917747646

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. … Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. … These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. …

In the scientific process, creativity is mostly associated with the generation of testable hypotheses and the development of suitable research designs. Data analysis, on the other hand, is sometimes seen as the mechanical, unimaginative process of revealing results from a research study. Despite methodologists’ remonstrations … it is easy to overlook the fact that results may depend on the chosen analytic strategy, which itself is imbued with theory, assumptions, and choice points. …

Researchers may understand this conceptually, but there is little appreciation for the implications in practice. In some cases, authors use a particular analytic strategy because it is the one they know how to use, rather than because they have a specific rationale for using it. Peer reviewers may comment on and suggest improvements to a chosen analytic strategy, but rarely do those comments emerge from working with the actual data set…. Moreover, it is not uncommon for peer reviewers to take the authors’ analytic strategy for granted and comment exclusively on other aspects of the manuscript. More important, once an article is published, reanalyses and critiques of the chosen analytic strategy are slow to emerge and rare … in part because of the low frequency with which data are available for reanalysis…. The reported results and implications drive the impact of published articles; the analytic strategy is pushed to the background.

But what if the methodologists are correct? What if scientific results are highly contingent on subjective decisions at the analysis stage? In that case, the process of certifying a particular result on the basis of an idiosyncratic analytic strategy might be fraught with unrecognized uncertainty … and research findings might be less trustworthy than they at first appear to be….. Had the authors made different assumptions, an entirely different result might have been observed….

… The analytic techniques chosen ranged from simple linear regression to complex multilevel regression and Bayesian approaches. The teams also varied greatly in their decisions regarding which covariates to include….

Figure 2 shows each team’s estimated effect size, along with its 95% confidence interval (CI). As this figure and Table 3 show, the estimated effect sizes ranged from 0.89 (slightly negative) to 2.93 (moderately positive) in odds-ratio (OR) units; the median estimate was 1.31. The confidence intervals for many of the estimates overlap, which is expected because they are based on the same data. Twenty teams (69%) found a significant positive relationship, p < .05, and nine teams (31%) found a nonsignificant relationship. No team reported a significant negative relationship. …

Table 3. Analytic Approaches and Results for Each Team

[70] Paper: “Observational Research Rigour Alone Does Not Justify Causal Inference.” By Keisuke Ejima and others. European Journal of Clinical Investigation, October 6, 2016. https://onlinelibrary.wiley.com/doi/10.1111/eci.12681

Observational designs remain useful in biomedical and behavioral research by allowing empirical investigations of exposures and their associations in populations experiencing all the vagaries of everyday life. Observational studies offer potential directions for randomized controlled experimental studies.

[71] Paper: “Resisting Rape: The Effects of Victim Self-Protection on Rape Completion and Injury.” By Jongyeon Tark and Gary Kleck. Violence Against Women, March 2014. https://journals.sagepub.com/doi/epdf/10.1177/1077801214526050

Page 270:

The impact of victim resistance on rape completion and injury was examined utilizing a large probability sample of sexual assault incidents, derived from the National Crime Victimization Survey … and taking into account whether harm to the victim followed or preceded self-protection (SP) actions. Additional injuries besides rape, particularly serious injuries, following victim resistance are rare. Results indicate that most SP actions, both forceful and nonforceful, reduce the risk of rape completion, and do not significantly affect the risk of additional injury. …

Finally, given the impossibility of experimental research on this topic, it should be noted that our findings are necessarily based on observed associations between victim actions and assault outcomes, thereby precluding definitive conclusions about causal effects.

[72] Book: Rutherford’s Vascular Surgery (8th edition, Volume 1). Edited by Jack L. Cronenwet and K. Wayne Johnston. Elsevier Saunders, 2014.

Chapter 1: “Epidemiology and Clinical Analysis.” By Loius L. Nguyen and Ann DeBord Smith.” Pages 2–14.

Observational Studies

There are two main types of observational studies: cohort and case-control studies. A cohort is a designated group of individuals that is followed over a period of time. Cohort studies seek to identify a population at risk for the disease of interest. After a period of observation, patients in whom the disease develops are compared with the population of patients who are free of the disease. Cohort studies are most often associated with epidemiology because they comprise many of the most prominent studies in the modern era. The classic example is the Framingham Heart Study (FHS), in which 5,209 residents of Framingham, Massachusetts, were monitored prospectively, starting in 1948.³ Much of our epidemiologic knowledge regarding risk factors for heart disease comes from the FHS.⁴ Although the FHS was initially intended to last 20 years, the study has subsequently been extended and now involves the third generation of participants. Cohort studies also seek to identify potential risk factors for development of the disease of interest. For example, if cigarette smoking is suspected in the development of peripheral arterial disease (PAD), smokers are assessed for the development of PAD from the beginning of the observation period to the end of the observation period. Because PAD does not develop in all smokers, and conversely, not all PAD patients are smokers, a relative risk (RR) is calculated as the ratio of the incidence of PAD in smokers versus the incidence of PAD in nonsmokers.

[73] Textbook: Principles and Practice of Clinical Research. By John I. Gallin and ‎Frederick P. Ognibene. Academic Press, 2012.

Page 226: “While consistency in the findings of a large number of observational studies can lead to the belief that the associations are causal, this belief is a fallacy.”

[74] Commentary: “Detecting Selection Bias in Observational Studies—When Interventions Work Too Fast.” By Ghulam Rehman Mohyuddin and Vinay Prasad. JAMA Internal Medicine, June 12, 2023. https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2805974

Early separation of the Kaplan-Meier curves is another mechanism for detecting residual confounding in observational studies. The Kaplan-Meier survival curve is a graphical representation of time-to-event end points and allows for maximal use of each participant’s time-related data. In a Kaplan-Meier analysis, participants contribute to the survival estimate until the event of interest occurs (eg, death, progression of disease) or until they are censored (eg, loss to follow-up or mandatory data lock). …

A limitation to this approach to detecting selection bias in observational studies is that it requires knowledge of the pathophysiology and natural history of a disease, the expected efficacy of the interventions, or other expertise in the subject of the study. Another is that when observational studies are published, Kaplan-Meier curves are often not included. Instead, there are tables showing relative risks or odds ratios for the comparisons between the groups of patients that are being observed; such tables do not show changes in the relative risks or odds ratios that may occur over time. To allow for the ascertainment of bias, such as imbalances between the characteristics of cohorts and residual confounding or confounding by indication, observational studies that examine a time-to-event end point or contain time-to-event data should report this information in a graphical form. When interventions appear to work too fast, the findings of a study may be too good to be true.

[75] Lecture: “Observational Studies – Basics, Agreement with Randomized Trials, Flawed Vaccine Studies.” Delivered by Dr. Vinay Prasad at the University of California San Francisco, November 3, 2023. https://www.youtube.com/watch?v=nlGiFTsaO7M

Time marker 41:30:

Eric Topol, he makes this mistake all the time. Here he is publishing that Covid-19 vaccine reduces the risk of long Covid.

Look at when the curves split! Covid-19 vaccine—it prevents you from getting long Covid before it can even prevent you from getting Covid! How is this plausible?

It’s not plausible. This is an observational study, and the reason it’s not plausible is that … [it] cures you of long Covid immediately. …

If interventions affect end points that cannot be plausibly related to the intervention, or if they work too fast, then I think that’s a fair sign that there is residual confounding in observational studies.

Here is Eric Topol on Paxlovid, which he thinks is so wonderful—85% reduction in death. And look at the curves split instantly! Paxlovid works even before you take it!

[76] Article: “Vaccines Stop Long Covid Immediately! Maybe 29 Days Before You Get It!” By Dr. Vinay Prasad, March 2, 2023. https://www.drvinayprasad.com/p/vaccines-stop-long-covid-immediately

A recent paper is out in BMJ Open; it is kind of funny. It is clearly preposterous. …

Simply put, the authors take a French dataset of long covid sufferers, and compare people who were vaccinated vs. those who were not with a sophisticated matching method. (Of course, it can’t correct for unmeasured covariates— but we don’t need to go there)

They ask: is vaccination associated with resolution of long covid symptoms? Resolution is defined as the symptoms vanish entirely. Here is the key Figure. …

Long covid starts to vanish immediately after vaccination! Vaccines work faster to stop long covid than they can stop the coronavirus. But wait, it actually works even sooner, because the questionnaire says PRIOR 30 days!

NOTE: See the footnote below for the study in question.

[77] Paper: “Efficacy of First Dose of Covid-19 Vaccine Versus No Vaccination on Symptoms of Patients with Long Covid: Target Trial Emulation Based on ComPaRe E-Cohort.” By Viet-Thi Tran and others. BMJ Medicine, February 27, 2023. https://bmjmedicine.bmj.com/content/bmjmed/2/1/e000229.full.pdf

Page 1:

Adult patients (aged ≥18 years) enrolled in the ComPaRe cohort before 1 May 2021 were included in the study if they reported a confirmed or suspected SARS-CoV-2 infection, symptoms persistent for >3 weeks after onset, and at least one symptom attributable to long covid at baseline. Patients who received a first covid-19 vaccine injection were matched with an unvaccinated control group in a 1:1 ratio according to their propensity scores. Number of long covid symptoms, rate of complete remission of long covid, and proportion of patients reporting an unacceptable symptom state at 120 days were recorded. …

By 120 days, vaccination had reduced the number of long covid symptoms (mean 13.0 (standard deviation 9.4) in the vaccinated group v 14.8 (9.8) in the control group; mean difference −1.8, 95% confidence interval −3.0 to −0.5) and doubled the rate of patients in remission (16.6% v 7.5%, hazard ratio 1.93, 95% confidence interval 1.18 to 3.14). Vaccination reduced the effect of long covid on patients’ lives (mean score on the impact tool 24.3 (standard deviation 16.7) v 27.6 (16.7); mean difference −3.3, 95% confidence interval −5.7 to −1.0) and the proportion of patients with an unacceptable symptom state (38.9% v 46.4%, risk difference −7.4%, 95% confidence interval −14.5% to −0.3%). In the vaccinated group, two (0.4%) patients reported serious adverse events requiring admission to hospital.

Page 7:

Our study had some limitations. Firstly, despite the use of robust methods and statistical techniques to make causal inferences from observational data, the intervention was not randomly assigned, and potential unmeasured confounders could have biased our results. For example, patients’ motivation to receive a covid-19 vaccine was not taken into account, although it might be related to their perception of their long covid symptoms. …

… patients in this study had not been vaccinated before their infection and long covid….

[78] Article: “Cause Versus Association in Observational Studies in Psychopharmacology.” By Chittaranjan Andrade. Journal of Clinical Psychiatry, August 26, 2014. https://www.psychiatrist.com/jcp/cause-versus-association-observational-studies-psychopharmacology/

In observational studies, causal explanations for identified associations may be supported in various ways. These include the presence of biological plausibility for the association, existence of a dependent relationship between the suggested dose-dependent relationship between the suggested cause and the effect of interest, and replication of the finding across studies.²⁹ However, none of these supports is foolproof.

[79] Article: “Dose-Response Relationship.” By Sydney Pettygrove (PhD). Encyclopædia Britannica. Accessed April 18, 2025 at https://www.britannica.com/science/dose-response-relationship

A dose–response relationship is one in which increasing levels of exposure are associated with either an increasing or a decreasing risk of the outcome. Demonstration of a dose-response relationship is considered strong evidence for a causal relationship between the exposure and the outcome.

[80] Paper: “The Dose-Response Relationship Between Cigarette Consumption, Biochemical Markers and Risk of Lung Cancer.” By MR Law and others. British Journal of Cancer, 1997. https://pmc.ncbi.nlm.nih.gov/articles/PMC2223525/pdf/brjcancer00188-0136.pdf

“There is an approximately linear relationship between the number of cigarettes per day that a person reports smoking and the age-specific risk of lung cancer—as consumption doubles, risk doubles.”

[81] Book: Rutherford’s Vascular Surgery (8th edition, Volume 1). Edited by Jack L. Cronenwet and K. Wayne Johnston. Elsevier Saunders, 2014.

Chapter 1: “Epidemiology and Clinical Analysis.” By Loius L. Nguyen and Ann DeBord Smith.” Pages 2–14.

Observational Studies

There are two main types of observational studies: cohort and case-control studies. A cohort is a designated group of individuals that is followed over a period of time. Cohort studies seek to identify a population at risk for the disease of interest. After a period of observation, patients in whom the disease develops are compared with the population of patients who are free of the disease. Cohort studies are most often associated with epidemiology because they comprise many of the most prominent studies in the modern era. The classic example is the Framingham Heart Study (FHS), in which 5,209 residents of Framingham, Massachusetts, were monitored prospectively, starting in 1948.³ Much of our epidemiologic knowledge regarding risk factors for heart disease comes from the FHS.⁴ Although the FHS was initially intended to last 20 years, the study has subsequently been extended and now involves the third generation of participants. Cohort studies also seek to identify potential risk factors for development of the disease of interest. For example, if cigarette smoking is suspected in the development of peripheral arterial disease (PAD), smokers are assessed for the development of PAD from the beginning of the observation period to the end of the observation period. Because PAD does not develop in all smokers, and conversely, not all PAD patients are smokers, a relative risk (RR) is calculated as the ratio of the incidence of PAD in smokers versus the incidence of PAD in nonsmokers.

[82] Paper: “Mendelian Randomization Analysis of the Causal Effect of Cigarette Smoking on Hospital Costs.” By Padraig Dixon and others. Nicotine & Tobacco Research, April 17, 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11494471/

Causal evidence on the effect of smoking on health care costs is also necessary for the robust evaluation of specific interventions that aim to prevent smoking and to treat its downstream consequences. Decision making by individual smokers may be improved with better information about the non-health consequences of smoking.¹²

However, establishing the causal effect of cigarette smoking on health care costs is challenging. Observed associations of smoking with health care costs may arise because smoking is indeed a cause of health care costs, because smoking is itself partly determined by health care costs, or because smoking is associated with causes or consequences of processes that influence health care costs. In general, it is not clear if the factors that may predispose an individual to smoke are themselves independent determinants of health care costs. For example, smoking tends to cluster with other behaviors known or suspected to affect health care costs, including high body mass index (BMI), poor diet, alcohol consumption, and low-physical activity.^13–17 Smoking is also heavily socially patterned; globally, lower socio-economic status groups are more likely to smoke than higher status groups^18,19 although this overall picture conceals variation over time and within regions. For example, in the first half of the 20th century, smoking prevalence was highest amongst higher socio-economic groups, but this pattern has now reversed.²⁰ Similar patterns of greater smoking prevalence amongst high-income groups are observed in some low-income countries.²¹ The point remains that the cooccurrence of smoking and socio-economic status is a challenge for conventional analyses.

Smoking is also more prevalent amongst groups defined by some health statuses. For example, smoking is more common amongst individuals with depression and schizophrenia.^22–25 These associations may affect all or some of smoking initiation, smoking intensity, and smoking cessation. Smoking also influences disease incidence (such as lung cancer) which in turn may prompt cessation. Smoking may reflect elements of self-medication²⁶ or a desire to control one’s weight.²⁷ Smoking may therefore be both a cause and consequence of health status and other circumstances.^28,29 Measured associations of smoking with health care cost may also partly reflect wider attitudes to risk tolerance, including impulsivity and behavioral disinhibition.

[83] Book: Multiple Regression: A Primer. By Paul D. Allison. Pine Forge Press, 1998.

Chapter 3: “What Can Go Wrong With Multiple Regression?” https://us.sagepub.com/sites/default/files/upm-binaries/2726_allis03.pdf

Pages 52–53:

When we estimate a regression model, we often interpret the coefficients as measuring the causal effects of the independent variables on the dependent variable. But what if the “dependent” variable actually affects one or more of the “independent” variables? If it does, the resulting biases can be every bit as serious as those produced by the omission of important variables. This problem— known as reverse causation—actually can be worse than the omitted variables problem because

• Every coefficient in the regression model may be biased

• It’s hard to design a study that will adequately solve this problem

Unfortunately, there’s rarely any information in the data that can help you determine the direction of causation. Instead, decisions about the direction of causation have to be based almost entirely on your knowledge of the phenomenon you’re studying. There are, in fact, several different ways to argue against the possibility of reverse causation. If the data come from a randomized experiment, then randomization assures us that the dependent variable isn’t influencing who gets the treatment. Often, the time ordering of the variables gives us a pretty clear indication of the causal direction. For example, we usually feel safe in supposing that parents’ educational attainment affects the educational attainment of their adult children, not the other way around. Even when there’s no time ordering, our knowledge of basic physical and biological processes sometimes gives us a pretty good idea of the causal direction. We feel confident, for example, that a man’s height might affect his social prestige, but social prestige couldn’t affect height.

With non-experimental data, most applications of regression analysis involve some ambiguity about the direction of causality. In such cases, the causal mechanism could run in either direction, perhaps in both.

[84] Report: “Improving Health and Social Cohesion through Education.” Organization for Economic Cooperation and Development, Center for Educational Research and Innovation, 2010. https://www.oecd.org/en/publications/2010/09/improving-health-and-social-cohesion-through-education_g1ghce50.html

Pages 31–33:

(a) Reverse causality

One source of endogeneity stems from the possibility that there is reverse causality, whereby poor health or low CSE [civic and social engagement] reduces educational attainment. Poor health in youth might interfere with educational attainment by interfering with student learning because of increased absences and inability to concentrate. It may also lead to poor adult health, thus creating a correlation between education and adult health. Similarly, low CSE such as lack of trust and political interest might also reduce educational attainment. For example, a family with low CSE might reduce their involvement with schools, which might lead to poorer student outcomes.⁷

The bias due to reverse causality can be re-cast as an omitted variable problem after considering timing issues. Since health and CSE tend to persist over time, past health or CSE can be an important determinant of current health or CSE. Thus, past health or CSE is an omitted variable in equation (1) which is captured by the error term. The extent to which omitting past health or CSE will lead to an omitted variable bias depends on the extent to which past health or CSE is also correlated with the included variable Education. Because the current stock of education depends on past decisions about investments in education, reverse causality generates a correlation between past health or CSE and the individual’s current stock of education.⁸ If the estimated coefficient picks up the effect of past health or CSE … will be biased towards overestimating the causal effect of education.

(b) Hidden third variables

The second source of endogeneity comes from the possibility that there might be one or more hard-to-observe hidden third variables which are the true causes of both educational attainment and health and CSE.⁹ In the context of the education–earnings link, the most commonly mentioned hidden third variable is ability.¹⁰ The long-standing concern in this line of research has been that people with greater cognitive ability are more likely to invest in more education, but even without more education their higher cognitive ability would lead to higher earnings (Card, 2001). More recently, non-cognitive abilities such as the abilities to think ahead, to persist in tasks, or to adapt to their environments have been suggested as important determinants of both education and earnings outcomes (Heckman and Rubinstein, 2001).

In the context of the education–health link, Fuchs (1993) describes time preference and self-efficacy as his favorite candidates for hidden third variables. People with a low rate of time preference are more willing to forego current utility and invest more in both education and health capital that pays off in the future (Farrell and Fuchs, 1982, Fuchs, 1982). A classic example is the Stanford Marshmallow Experiment in which 4 year-olds were given the choice between eating the marshmallow now or waiting for the experimenter’s return and getting a second marshmallow. When these children were tested again at age 18, Shoda and others (1990) found a strong correlation between delayed gratification at age 4 and mathematical and English competence. Similarly, people with greater self-efficacy, i.e. those who believe in their ability to exercise control over outcomes, will be more likely to invest in schooling and health. Most studies of the schooling–health link use data sets that do not contain direct or proxy measures of time preference and self-efficacy. Consequently, these variables are typically omitted when estimating equation (1). The resulting omitted variable bias again implies that … will be biased towards overestimating the causal effect of education on health.

[85] Paper: “Retrospective Analyses of Large Medical Databases. What Do They Tell Us?” By Richard a. Ward and Michael E. Brier. Journal of the American Society of Nephrology, February 1999. https://jasn.asnjournals.org/content/10/2/429.short

The clinical study of human disease is complicated by interdependent variables, and powerful analytical tools are necessary to establish causal relationships. Prospective studies can be randomized and blinded to the investigators. These techniques protect prospective trials, somewhat, from the problems of bias in the study design and confounding by codependent variables. However, prospective clinical trials are difficult to perform, may require extended duration for adequate observations of human diseases, and are expensive to organize and perform. Retrospective studies may take less time and are less expensive than prospective studies because the data have already been measured. However, retrospective studies are susceptible to bias in data selection and analysis. Furthermore, confounding variables may go unrecognized because of inadequate knowledge of how they interrelate with the outcome of interest. Because of these limitations, retrospective data analysis may show associations among variables, but rarely establishes causal relationships.

Dialysis has achieved widespread clinical acceptance because its efficacy is undisputed and the outcome without therapy is obvious. Patients with end-stage renal disease (ESRD) die of the complications of uremia, unless they are dialyzed or receive a renal transplant. Because the consequence of not dialyzing a patient with ESRD is so clear, dialysis was never subjected to the rigors of a prospective, randomized clinical trial.

Related