For folks interested in Medical Reversal
In the wake of the uspstf decision on aspirin for primary prevention a lot of people have come out with all sorts of theories about how medical science ought to progress what went wrong in this instance i think many of them are missing the mark so let me walk you through my thinking on this i’m going to talk about medical reversal and we’ve been studying this for
Quite some time now maybe over 10 years at this point and we’ve written a book i’ll put a link in the description ending medical reversal which is literally on this topic what is medical reversal well to understand it you need to first know how science should normally proceed the way medical science should work is and of course medical science is just one small
Subset of all of science the way medical science should work is we are in the business of developing tests screening interventions such as drugs devices procedures surgeries that are all supposed to make you live longer live better and the way it should work is a scientist develops a hypothesis they develop a model of how the body works a theory of what might make it
Better they develop a lot of circumstantial evidence that supports that theory corroborates that theory that this should work and then they go ahead and implement it nope actually they do the key study they do a definitive study that asks whether the routine application of this test this pill this drug this surgery actually makes you live longer live better and in
General that type of study is a randomized control trial and you want it to have adequate power adequate blinding you want it to measure the endpoint you really care about living longer living better you need all that and then you go ahead and make the leap and even with all this stringent safeguards there’ll still be some error rate they’ll still get some things
Wrong that’s the nature of statistical science however that error rate will be quite low but what happens instead i think too often in biomedicine we get seduced by the mechanism we get seduced by the novelty there are a lot of people who are going to make a lot of money if you sell this product and so they bring that product out prematurely they develop the model
They develop the circumstantial evidence and then they just debut the product without doing that gold standard testing we just saw this with ajikanamab is it gonna make people with alzheimer’s live longer live better i don’t know does it change beta amyloid plaque in the brain sure does that correlate with that maybe but let’s just go ahead and debut it and start
Paying for it and start implementing it and that is a perennial pitfall in biomedicine and it’s also been a perennial pitfall even in soros cove too we left before we had evidence in so many fronts now part of that might be reasonable but part of that is also unjustified if you don’t simultaneously try to remedy your lack of knowledge so there’s one phenomenon
Called replacement that’s where new practices replace the next thing and then there’s medical reversal where you debut the practice prematurely it exists often for decades and then finally some brave investigator comes along and puts it to a rigorous test and it often fails it doesn’t work it is reversed it’s not that it was replaced with something better it’s that
It was no better than what we did before why does reversal happen well sometimes it’s because you debuted the practice based on an underpowered phase 2 study it didn’t measure the end point you cared about it didn’t have the power and then later you did a randomized phase 3 trial with adequate power on the endpoint you actually care about and it doesn’t work because
Underpowered phase 2 studies are both notorious for false negative results but also false positive results and here’s a good example lar truvo that sarcoma drug was debuted under accelerated approval based on an underpowered phase two and lo and behold the phase three trial didn’t work at all and was yanked from the market the next common pitfall is extrapolating
Single center randomized control trial results to a multi-center context good examples of this are manny rivers early goal directed therapy tight glycemic control in the netherlands these studies these practices they appeared to work in some single centers with prominent spokespeople but when you tried to scale that up to a multi-center study the results evaporated
It didn’t work some people say that this is just the normal way science proceeds well that’s not right because the normal way science proceeds would be to take that pilot study and test it and then implement it that would be a logical way to do it but to implement it first and then conduct the definitive trials years later that is a bit problematic in my mind the
Other thing it points to is that um some people say that you know it did work but it didn’t work science is just evolving actually it may have worked at that one center but that was never the relevant question the question was always could other places implemented and get the same results and the answer is only tested by the multi-center trial and shows it doesn’t
The other sort of classic pitfall we fall into is extrapolating beyond a study a study showed something work and severe illness people with certain inclusion criteria we start extrapolating that to people beyond that initial inclusion criteria and lo and behold it doesn’t work as well or doesn’t work at all in some of those groups may even have harms outweighing
Benefits the other way it doesn’t work is randomized control trials are done but they don’t measure what you care about they didn’t measure the right endpoint they used a surrogate endpoint they had crossover when you didn’t want it they didn’t have it when you did want it they used a bad control arm they have poor prose protocol care i’ve written about this
Extensively in the book malignant and those are just a whole class of examples where even randomized trials can mislead the other sort of classic way we fall into these pitfalls is um we embrace something based on uncontrolled or historical evidence we didn’t have randomized data showing a benefit we just had 50 people’s experience tested against historical benchmarks
We thought it was better we never did the randomized study we just embraced it the problem with that is historical benchmarks are often historical for a reason lots of other things were different in the past we’re going to improve upon historical benchmarks doesn’t mean the intervention actually works but the biggest class of reasons the biggest reason we get so
Much flip-flopping so much medical reversal is we embarked on practices based on nothing other than mechanistic science and observational retrospective studies we were so eager enthusiastic we just embarked on it and human beings are quite good at doing that we’re an optimistic species we love to embrace new things we hope we can make our condition better but when
It comes to biology that’s very very difficult we often stutter we often fall we often fail and that’s the biggest class of the reasons so let’s talk about aspirin what’s this new aspirin study show i think this aspirin flip-flop is a little bit different than these other things because it’s actually probably because the underlying substrate has actually changed
The initial randomized control trials of aspirin and prevention were successful but that was a different person it was a person who was an older person smoker thin and aspirin seemed to work in that population there wasn’t a lot of statin use fast forward a few decades now people are higher bmi the more likely to have diabetes they’re more likely to be on a statin
Does aspirin work in this setting and in 2019 we wrote that paper should medical therapies come with an expiration date because many of these changing patterns in our society may make some therapies that used to work no longer work and that’s a little bit different than reversal it wasn’t that we got it wrong it was that the underlying substrate changed but the
Failure of course is not continually reassessing our evidence base not running these studies testing making sure it works in different contexts in many other settings it’s likely that a practice that used to work no longer works but we just don’t know it because we’ve never tested it again so aspirin is a different sort of class than medical reversal but medical
Reversal certainly is real medical reversal far exceeds the uncertainty of statistics if you ensured adequate well-controlled randomized studies before adopting costly new medical practices you would dramatically reduce the rates of medical reversal it wouldn’t be nearly what it is today so when i see people talk about this and they say you know manny rivers
Didn’t get it right didn’t get it wrong okay sure he didn’t get it wrong for his hospital but no one really cared about just his hospital we cared about all the other hospitals and in that sense the study was not very informative it didn’t predict what happened there somebody might say um you know some theory based on pathophysiology like the swan gans catheter it
Wasn’t wrong well yeah actually when you did a randomized controlled trial the routine application of swan gans you couldn’t use it to leverage better outcomes and over and over again we see the same problem the problem of course is with pills and iv drugs you can adopt these based on insufficient evidence i think we see that with the exondus the duchess muscular
Dystrophy drug we see this with educamab but it also goes beyond that it goes beyond to multi-component procedural quality safety metrics benchmarks uh incentive structures all these things that are done to leverage better health outcomes seldom if ever come with randomized data if i would articulate what i think the biggest failure of the pandemic was was that
We did do a good job of running some randomized control trials of therapies we probably had too much off-label use while those trials were running just excessive off-label use that siphoned patients away from those studies you can just think about that the debacle of the mount sinai investigators giving thousands of people anticoagulation outside of a randomized
Study prior to the randomized studies when they could have easily joined the randomized studies and actually answered the question much sooner sometimes the evidence came a little bit too late in that case you know most of uh of of the patients had already been in hospital and out of hospital before we started to learn with some of these new randomized trials which
Population benefits from aggressive anticoagulation and which benefits from usual standard of care anticoagulation we did a good job with some of the therapies but others like convolus and plasma we treated way too many patients without getting the randomized evidence on where that might help if at all dexamethasone was an early success story but i think people
Didn’t understand and they were somehow reluctant to embrace the early pre-print results um but when it comes to the non-pharmacological interventions we failed catastrophically the failure wasn’t that we lept on some things early on in the pandemic without credible evidence that wasn’t the failure at all the failure was that while we were leaping onto things we
Actively poisoned and didn’t run cluster randomized controlled trials we have just a paucity of them uh we’ve talked about bangladesh on this channel i think bangladesh shows many of the challenges it was done but uh you know it came very late i mean it came 18 months into the pandemic it does show surgical masks was a winner but cloth mask didn’t win it all but
Yet cough masks were what we used to and still mandate actually it’s still the mask that most places are mandating they’re not mandating the surgical mask bangladesh of course applied to a place with essentially no pre-existing immunity and no vaccination and we’re extrapolating that to places like san francisco with 80 vaccination or extrapolating that to places
In the south with high vaccination plus maybe even high zero prevalence we don’t know we’re not good at documenting that so these are all ways in which you go beyond the available evidence and it’s easy to believe that when you go beyond the available evidence you’re making the world a better place the reality is often the odds are stacked against you and that
These interventions are not going to work so there are several categories of way medical science progresses one is replacement a drug came along it was better than no drug at all and then a new drug comes along it’s better than the older drug when we go in order steady step-wise incremental progress i think that’s how many of us conceptualize biomedicine there’s
Another pathway that’s often very common it’s where we weren’t doing something we were desperate we were wish we could have something and we had some pathophysiology some theory and we jumped on something new years elapse before brave investigators conduct randomized control trials and turned out that something new was actually no better than doing nothing at all
Or doing whatever we did before and that’s a medical reversal and then there’s this other phenomenon that happened with aspirin which was that we were doing something we had good evidence very likely it did help but then the people changed over time it was fundamentally different group of people because of all these shifting changes in society but we didn’t reassess
The evidence with the sort of time span we need maybe we’re lucky we did it at all and when we did it found it didn’t work and then we had the flip flop the last one is more forgivable in my mind than the center one than reversal reversal is not really forgivable for me because you know i think for maybe 50 60 years that you really need to develop robust evidence
Um and i think the real key insight into reversal why it’s so important is that there have been many things in biomedicine that were really plausible that we really thought worked that people said i will not randomize you to not doing it because i’m so confident it does work that when you actually test it explicitly it did not work at all in fact increased death
And the classic example is the cardiac antiarrhythmic suppression study suppression trial cast in the new england journal of medicine which actually had an increased death rate from those class 1c antiarrhythmics we detail many of these examples and many more in the book ending medical reversal we provide a coherent i think framework philosophical framework for
How to think about medical technologies science is complicated science will always have one step forward two steps back two steps forward one step back sorts of moments but medicine is the application of science in pursuit of human health and we can minimize the rate of these backward steps if we set some clear standards of evidence and i think knowing the history
Of the flip-flops of medicine is important because it allows you to have some perspective about medical evidence and in this pandemic in particular i think people who did not have this perspective were eager to champion interventions based on mechanistic science retrospective observational data confounded county by county retrospective data et cetera et cetera
They didn’t really understand that these types of data have such poor credibility it’s almost like no data at all in many cases it’s really almost at the level of credibility of nothing at all nothing at all has no credibility and some of these studies are not much better than no credibility at all you need to do credible studies if you wish to consider yourself
An enlightened species and i think that’s what i struggle with so aspirin not a medical reversal but an important lesson medical reversal very common very problematic sort of classic pitfalls that occur in the path to medical reversal excision circumstances yes if you’re in a dire straits you can definitely do something for some period of time without medical
Evidence i think no one will fault you for that for things that are dire for rare for new things but simultaneously when those things persist you have some commitment developing robust evidence and that commitment is in part because you know what you’re doing is very likely to not actually work i mean that’s the reality and then the final thing to say is the
Pre-test probability the pre-protest probability the probability that anything someone dreams up will make people live longer live better it’s very very low in biomedicine it’s very very low not because i say it’s very very low it’s very very low because biology has made it so we are an exquisitely adapted species it is very difficult to improve upon that adaptation
Even in sickness and in health there are a lot of people who have the compulsion the will to believe that we can do something to interdict these processes and improve our outcomes and fate that belief is often erroneous exaggerated false it is a natural human compulsion and so we see it over and over again people who work in fields where they have more control
Over their domain people who work in technology they often come into biomedicine and they have so many questions like why can’t we just do this to get this problem to go away but the truth is the biology the complexity of the body is far more complex than your phone or your tv your car your camera it really is not fully explicable in terms of fundamental parts
And components it cannot be reduced uh at least not yet and as long as that’s the case you need broad scale empirical large simple randomized control trials to filter the signal from the noise this is the single most important lesson in biomedicine so if you like this video like subscribe comment leave a message read the book ending medical reversal i think it
Will give you a real perspective of flip-flops in biomedicine these are just some thoughts i had when i was reading people talk about aspirin and i don’t think they’ve thought too much about how medical progress should occur so until next time
Transcribed from video
ASPIRIN & Medical Flip Flops | Reversal, Replacement, Evidence Expiration Dates | A doctor discusses By Vinay Prasad MD MPH
