When research can’t be replicated and unequal access to psychiatry and broadband

By Tom Insel, MD, and Matthew Hirschtritt, MD, MPH

Happy 2022, Research Roundup readers. For our first edition of the year I am delighted to welcome Dr. Matthew Hirschtritt who will be joining me to find and summarize the best science stories of the week. We start with a report from a massive project on reproducibility in research. It comes from cancer biology but brings an important ­– if sobering – message to all of us in health. Also: new studies on the child psychiatry workforce, the importance of eviction protection for mental health, and the largest review ever on violence and schizophrenia. Read on!   

When Research Can’t Be Replicated

Reproducibility is the ground truth of science. The scientific enterprise rests on the fundamental premise that an independent researcher replicating a carefully controlled experiment will find similar if not identical results. No surprise then that we are still hearing the echoes of papers that scientists at Amgen and Bayer published a decade ago reporting that many findings from the cancer biology literature could not be replicated. These reports were followed by a stunning failure to replicate more than half of the results from 100 psychology papers published in the four highest-ranking journals.

To explore this problem, the Center for Open Science and Science Exchange, two nonprofit organizations, worked with the journal eLife to launch the Reproducibility Project-Cancer Biology (RPCB). With careful planning and often consultation with the original scientists, the RPCB tried to replicate 193 experiments from 53 high-profile papers in cancer biology.

The results of this multi-year, multi-site project were reported at the end of 2021 — and they were sobering. Almost every paper lacked sufficient detail to permit an accurate replication of key experiments. Only 50 of the 193 experiments could be tested for reproducibility, and of those, just 40% of positive effects and 80% of negative effects could be replicated. Effect sizes in the replication study were smaller in 92% of positive cases. Bottom line: only a small fraction of the findings from high-profile papers in cancer biology could be reproduced.

Will studies in mental health research fare better? Basic cancer research might be a best-case scenario for reproducibility: It is generally performed in standardized cell lines and uses commercial reagents and quantitative outcomes like rates of cell division or cell death. Mental health research is in people, with a range of non-standardized interventions and outcomes that are often subjective. Is there any hope that findings from high-profile psychology and psychiatry journals can be reproduced? And does a lack of reproducibility condemn the research enterprise?

The editors of eLife published a helpful commentary that ends with these words: “The aim of the RPCB was not to find papers that were flawed or faulty, and a failure of the team to replicate an experiment does not mean that the original was wrong (and, likewise, a successful replication does not guarantee that the original was correct)…” It’s an important point. Biology is not physics. Variation is inherent in biological systems. A failure to replicate is not evidence of misconduct nor should it leave us cynical about published research. It simply reminds us that individual studies are simply single points on a graph. To paraphrase a statement about history, it is the arc of science that bends towards truth.     

Science is an iterative, tedious process that is 1% inspiration and 99% perspiration. Breakthrough findings are often less than they claim; eye-catching science may not be reproducible. Replication studies, which receive little funding and less acclaim, are a vital part of the transition from research to knowledge. Research Roundup will endeavor to summarize the most relevant, surprising, and sometimes inconvenient results in the peer-reviewed literature, especially results from high-profile journals. We will also remind readers that eye-catching science needs to be replicated. Reproducibility is our ground truth.

Investigating the replicability of preclinical cancer biology and Reproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology  eLife, Dec 7, 2021

---

Kids’ Mental Health Crisis: Who Ya Gonna Call?

The COVID-19 pandemic has worsened a mental health crisis among U.S. youth that started at least a decade ago. From 2009 to 2019, the proportion of high school students reporting persistent feelings of sadness or hopelessness increased by 40% and suicide rates among youth ages 10-24 increased by 57%. Now Covid has added fuel to the fire. In early 2021, emergency department visits for suspected suicide attempts were 51% higher for adolescent girls and 4% higher for adolescent boys compared to the same period in early 2019. Overdose deaths and suicides in children and youth – so-called deaths of despair – greatly outnumbered Covid-19 deaths. The need for mental health services for kids has never been higher.

In this context, RAND Corporation researcher Ryan McBain, PhD, MPH, and colleagues asked a question: How many U.S. youth have no access to virtual or in-person psychiatric services, and where do they live? They combined data from the Health Resources & Services Administration’s Area Health Resources Files and the U.S. Census to estimate the number of child and adolescent psychiatrists in each county and the number of child and adolescent psychiatrists per 100,000 youth (ages 5-19 years). To quantify access to internet broadband coverage, the researchers used data from the U.S. Department of Commerce to identify counties with poor access to high-speed internet.

As of 2018, nearly 10% of youth (more than 6 million) had no access to a child and adolescent psychiatrist or high-speed internet. In 2,324 counties (74%), there wasn’t a single child or adolescent psychiatrist. And while 69% of urban counties had high-speed internet, only 34% of rural ones did. Rural counties, on average, had 12% fewer child and adolescent psychiatrists per 100,000 youth than urban ones. The wealthiest 25% of counties – where median household income topped $58,850 – had 9.6 child and adolescent psychiatrists per 100,000 youth and 66% of households had access to high-speed internet. But the poorest 25% of counties ­– where median household income was $43,680 – had only 1.7 child and adolescent psychiatrists per 100,000 youth, and just 32% could adequately access high-speed internet.

These results highlight the need to identify ways to expand the reach of child and adolescent psychiatric services to underserved communities. Expanding broadband access to rural areas would help. Incentivizing recent psychiatry grads to practice in underserved areas might also level the playing field. The pandemic has reminded us that reliable, high-speed internet access is a social determinant of health; not having it is increasingly akin to experiencing housing instability and food insecurity. The U.S. needs to bridge the digital divide and provide mental health to all U.S. youth, regardless of where they live.

Ongoing Disparities in Digital and In-Person Access to Child Psychiatric Services in the United States Journal of the American Academy of Child and Adolescent Psychiatry, Dec 20, 2021

---

Housing Stability is Important for Mental Health During COVID-19

A safe and stable home is a prerequisite for mental health. Housing instability – which can include having trouble paying rent, overcrowding, moving frequently, living with relatives, or spending the majority of one’s income on housing – is associated with higher levels of depression and anxiety. In September 2020, the CDC issued a federal moratorium on evictions “to prevent the further spread of COVID-19” and the order was renewed multiple times before being struck down by the US Supreme Court in August 2021.

Before the CDC’s action, states created their own standards. Some imposed strong regulations to keep residents in their homes; others offered only tepid protections. This situation, like many during this pandemic, created a natural experiment: Did residents in states with stronger eviction protections experience better mental health?

Kathryn Leifheit, PhD, MSPH, a social epidemiologist at UCLA, and colleagues used data from 2317 renters with household incomes less than $75,000 who took part in the Understanding Coronavirus in America Survey between March 10 and September 3, 2020, the six months that preceded the federal eviction moratorium. Each state was classified according to the strength of their moratorium: none, weak (e.g. eviction notices could be filed but hearings and enforcement was halted), or strong (all stages of the eviction process were blocked). Using a four-item health questionnaire, the researchers identified renters who reported moderate-to-severe mental distress.

Renters living in states with weak or no eviction protections moratoriums had similar rates of moderate-to-severe mental distress. The results did show a connection: In states with few or no protections, 17% to 17.7% of renters, respectively, suffered from moderate to severe depression, compared with15.5% in states with strong moratoriums. That 2.2% reduction might seem minor, but it represents a relative reduction in prevalence of 12.6%, translating to millions of people with lower distress.

Deeper dives into the data found some other subtle differences: For example, the effects of strong moratoriums were even more pronounced for renters living in high-cost areas. After the federal moratorium was lifted, states including California and New York chose to extend their protections. But millions of other Americans now have little or no protection against becoming homeless. This study shows the impact of housing policy on their mental health.

Variation in State-Level Eviction Moratorium Protections and Mental Health Among US Adults During the COVID-19 Pandemic  JAMA Network Open Dec 1 2021

---

Schizophrenia and Violence: Is There an Association?

Nearly every anti-stigma campaign reminds us that people with mental illness are more likely to be the victims than the perpetrators of violence. That may indeed be true for people with a mental illness who are in treatment. But what about people with a psychotic illness without treatment? When symptomatic, they are, by definition, irrational. Are they more likely to be violent?

To answer this question, Daniel Whiting and his colleagues at Oxford University reviewed 24 studies from 15 countries across four decades with over 50,000 people who’d been  diagnosed with schizophrenia. Their conclusion is clear: Relative to the general population, men with schizophrenia were 4.5-fold more likely to be violent; women with schizophrenia were 10.2-fold more likely to be violent. The absolute risk – that is, the likelihood that an individual with schizophrenia would show interpersonal violence – ranged from one in four for men to one in twenty for women.

These results may seem to confirm the association of a psychotic mental illness with violence, but it is important to recognize that treatment breaks that association. This new report – the largest meta-analysis ever undertaken looking at violence in people with schizophrenia ­– did not investigate the role of treatment. But a 2020 study from Sweden found a near 90% reduction in risk of violence among people with psychotic disorders treated with the antipsychotic clozapine.

The message is not just that people with psychotic illnesses are more likely to be violent, but that treatment is effective at reducing this risk to both their safety and the safety of others. If advocates want to deliver an accurate message, they need to acknowledge the risk of violence while proclaiming the importance and effectiveness of treatment. For some, the stigma against treatment may be greater than the stigma against those with these disorders.

Association of Schizophrenia Spectrum Disorders and Violence Perpetration in Adults and Adolescents from 15 Countries A Systematic Review and Meta-Analysis JAMA Psychiatry, Dec 22, 2021

---

Tom Insel, MD, is a psychiatrist, neuroscientist, and former director of the National Institute of Mental Health (NIMH). He is a donor to MindSite News and chair of its Editorial Advisory Board. Dr. Insel’s financial conflict of interest statement, which includes equity and advisory roles in several early-stage mental health technology companies, can be found here.

Matthew Hirschtritt, is a clinical psychiatrist with The Permanente Medical Group, Inc.; a mental health services researcher with the Division of Research, Kaiser Permanente Northern California; and an assistant program director of the Kaiser Permanente Oakland Adult Psychiatry Training Program. His research work has covered diverse topics, ranging from pharmacotherapy for childhood mental disorders, to the epidemiology of Tourette syndrome, the needs of court-involved youth, and the mental health correlates of excessive alcohol use. His current research focuses on identification and treatment of patients with first-episode psychosis, as well as implementation of a telehealth-based mental health evaluation and referral program.