ABSTRACT: Abstract Most modern knowledge is not science. The physical sciences have successfully validated theories to infer they can be used universally to predict in previously unexperienced circumstances. According to the conventional conception of science such inferences are falsified by a single irregular outcome. And verification is by the scientific method which requires strict regularity of outcome and establishes cause and effect. -/- Medicine, medical research and many “soft” sciences are concerned with individual people in complex heterogeneous populations. These populations (...) cannot be tested to demonstrate strict regularity of outcome in every individual. Neither randomised controlled trials nor observational studies in medicine are science in the conventional conception. Establishing and using medical and other “soft science” theories cannot be scientific. It requires conceptually different means: requiring expert judgement applying all available evidence in the relevant available factual matrix. -/- The practice of medicine is observational. Prediction of outcomes for the individual requires professional expertise applying available medical knowledge and evidence. Expertise in any profession can only be acquired through experience. Prior cases are the fundament of knowledge and expertise in medicine. Case histories, studies and series can provide knowledge of extremely high reliability applicable to establishing reliable general theories and falsifying others. Their collation, study and analysis should be a priority in medicine. Their devaluation as evidence, the failure to apply their lessons, the devaluation of expert professional judgement and the attempt to emulate the scientific method are all historic errors in the theory and practice of modern medicine. (shrink)

As a way to make medical decisions, Evidence-Based Medicine (EBM) has failed. EBM's failure arises from not being founded on real-world decision-making. EBM aspires to a scientific standard for the best way to treat a disease and determine its cause, but it fails to recognise that the scientific method is inapplicable to medical and other real-world decision-making. EBM also wrongly assumes that evidence can be marshaled and applied according to an hierarchy that is determined in an argument by authority to (...) the method by which it has been obtained. If EBM had valid theoretical, practical or empirical foundations, there would be no hierarchy of evidence. In all real-world decision-making, evidence stands or falls on its inherent reliability. This has to be and can only be assessed on a case-by-case basis applying understanding and wisdom against the background of all available facts—the "factual matrix." EBM's failure is structural and was inevitable from its inception. EBM confuses the inherent reliability and probative value of evidence with the means by which it is obtained. -/- EBM is therefore an ad hoc construct and is not a valid basis for medical decision-making. This is further demonstrated by its exclusion of relevant scientific and probative real-world decision-making evidence and processes. It draws upon a narrow evidence base that is itself inherently unreliable. It fails to take adequate account of the nature of causation, the full range of evidence relevant to its determination, and differing approaches to determining cause and effect in real-world decision-making. EBM also makes a muddled attempt to emulate the scientific method and it does not acknowledge the role of experience, understanding and wisdom in making medical decisions. (shrink)

Medicine, like law, is a pragmatic, probabilistic activity. Both require that decisions be made on the basis of available evidence, within a limited time. In contrast to law, medicine, particularly evidence-based medicine as it is currently practiced, aspires to a scientific standard of proof, one that is more certain than the standards of proof courts apply in civil and criminal proceedings. But medicine, as Dr. William Osler put it, is an "art of probabilities," or at best, a "science of uncertainty." (...) One can better practice medicine by using other evidentiary standards in addition to the "scientific." To employ only the scientific standard of proof is inappropriate, if not impossible; furthermore, as this review will show, its application in medicine is fraught with bias. Evidence is information. It supports or undermines a proposition, whether a hypothesis in science, a diagnosis in medicine, or a fact or point in question in a legal investigation. In medicine, physicians marshal evidence to make decisions on how to best prevent, diagnose, and treat disease, and improve health. In law, courts decide the facts and render justice. Judges and juries assess evidence to establish liability, to settle custody and medical issues, and to determine a defendant's guilt or innocence. (shrink)

There is a middle ground of imperfect knowledge in fields like medicine and the social sciences. It stands between our day-to-day relatively certain knowledge obtained from ordinary basic observation of regularities in our world and our knowledge from well-validated theories in the physical sciences. -/- The latter enable reliable prediction a great deal of the time of the happening of events never before experienced. The former enable prediction only of what has happened before and beyond that of educated guesses which (...) may sometimes prove right and others not when we test them. -/- The imperfection of our knowledge between those limits is a consequence of complexity. -/- Reductionist empiricism fails when faced with complexity but we all still have to live in a complex world where reductionist science cannot help us devise reliable theories from which we can predict the behaviour of the world around us. We cannot predict reliably. We can only monitor actively and manage actively the world around us if we want to try to control our environment. We have a limited prediction horizon. -/- Science and empiricism work well if we want to send Voyager I and II over 3 billion miles away but not for most other things. -/- In medicine how and when to apply probabilistic, conjectural, incomplete medical theories and explanations requires professional expertise, intuition and judgement (non scientific knowledge) which is essential. Medical diagnosis is a skill of predicting from knowledge of what has happened before that the past will recur in the current patient applying expertise and intuition from knowledge and experience of prior cases and probabilistic medical research and theory. The physician is left to an educated guess as to how the future will develop for the particular individual patient the focus of his or her current attention. (shrink)

Eyal Shahar’s essay review [1] of James Penston’s remarkable book [2] seems more inspired playful academic provocation than review or essay, expressing dramatic views of impossible validity. The account given of modern biostatistical causation reveals the slide from science into the intellectual confusion and non-science RCTs have created: “…. the purpose of medical research is to estimate the magnitude of the effect of a causal contrast, for example the probability ratio of a binary outcome …” But Shahar’s world is simultaneously (...) not probabilistic, but of absolute uncertainty: “We should have no confidence in any type of evidence ….. We should have no confidence at all”. Shahar’s "Causal contrast" is attractive. It seems to make sense, but bypasses in two words the means of establishing causation by the scientific method. This phrase assumes a numeric statistically significant “contrast” is causal rather than a potential correlation requiring further investigation. The concept of “causal contrast” is a slippery slope from sense into biostatistical non-science. This can be illustrated with an hypothetical RCT where 100% of interventions exhibit a posited treatment effect and 0% of placebo controls. Internal validity is seemingly quite reasonably assumed satisfied (common-sense dictating the likelihood of an awesome magnificent fraud, bias or plain error of the magnitude required is infinitesimal). Scientific method appears satisfied. The RCT demonstrates: (1) strict regularity of outcome in the presence of posited cause; (2) the absence of outcome in its absence and (3) an intervention (experiment) showing the direction of causation is from posited cause to posited effect. Now travel further down the slope from science. Assume 50% of interventions and 0% of controls are positive. We compromise scientific method, but justify this by assuming a large subgroup which we say surely must on these figures be exhibiting the posited treatment effect. But what of 10% of interventions and 9% of placebo controls exhibiting the posited treatment effect? Our biostatistician says the 1% “causal contrast” is statistically significant. But we have: (1) minimal evidence of regularity; (2) the posited outcome irrespective of presence of posited cause and (3) our intervention is at the highest equivocal in demonstrating any form of causation. This is not science. It is, however, where biostatistics has unthinkingly taken us, as Penston has shown comprehensively [2]. We, the audience of published medical research, are now for the 10% / 9% example well down the slope from science. An unattractive hypothesis results requiring numerous assumptions similar to these:- "There is a 'contrast' which is ‘causal’, albeit the method employed is not scientific. An effect of the intervention has been observed in a very small subgroup. This subgroup is susceptible to treatment. The similar number of placebo controls exhibiting the outcome sought is irrelevant, because the 1% difference between intervention and controls is statistically significant. The statistical analysis is valid and reliable. The RCT’s internal validity is sufficiently satisfied. No funding or bias or fraud has affected the results or their analysis.” As Penston notes: “Confirming and refuting the results of research is crucial to science …. But … there’s no way of testing the results of any particular large-scale RCT or epidemiological study. Each study … is left hanging in the air, unsupported.” It gets worse. To identify a rare serious adverse reaction of a frequency of 1:10,000 can require a trial of 200,000 or larger split between controls and interventions. This is not done. But for every 100 who prospectively benefit from the intervention, 9,900 also receive it. And for every 100 benefiting one person (who likely gains no benefit) will suffer a serious unidentified adverse reaction. This is also without taking account of more common adverse reactions whether serious or otherwise. References [1] Shahar, E. (2011) Research and medicine: human conjectures at every turn. International Journal of Person Centered Medicine 1 (2), 250-253. [2] Penston, J. (2010). stats.con: How we’ve been fooled by statistics-based research in medicine. London: The London Press, UK. (shrink)

This exchange of correspondence with a critic provides helpful explanations of aspects of the paper "On Evidence, Medical and Legal" by the same authors.