SfN Takes Steps to Tackle Scientific Rigor Concerns

As is now widely known, a series of high-profile journal articles have raised concerns about difficulties in replicating a significant number of scientific studies, many involving preclinical research. Reports have subsequently appeared in the lay press echoing these concerns, and this issue has been raised in congressional hearings about science funding. The focus of these questions is not on scientific misconduct, but rather on issues such as experimental design, full reporting of methods and results, and the growth of incentives in academia and scientific publishing that might encourage haste and discourage both replication studies and the reporting of negative results.

Recognizing the centrality of scientific rigor both to progress in our field and to ensuring the public’s faith in science, the Society for Neuroscience established the Scientific Rigor Working Group in the fall of 2013. The working group was tasked with determining the scope of such issues within our community and, based on its findings, recommending proactive and constructive strategies to improve scientific practices. In addition, SfN President Steve Hyman has made efforts to advance scientific rigor an area of focus during his term. 

“SfN is uniquely positioned to bring together neuroscientists, funding agencies, and publishers to think deeply about this problem and to brainstorm solutions, and to influence the dissemination of those solutions across the field,” said Rita Balice-Gordon, a member of the working group and head of integrative neuroscience and neuroinflammation at Pfizer.

Working group co-Chair Emanuel DiCicco-Bloom explained that addressing concerns about reproducibility will require efforts in identification of guidelines for rigorous science, training and professional development, journal review and publication policies, and public communication. SfN has committees and staff dedicated to many of these areas, putting the Society in an ideal position to tackle the issue, said DiCicco-Bloom, a professor of neuroscience and cell biology and pediatrics at Rutgers’ Robert Wood Johnson Medical School.

In forming the working group, the Society aimed to bring together a diverse collection of SfN members with the expertise and breadth of knowledge to identify and conceptualize the relevant issues and responses. The group is composed of representatives from Council and nine SfN committees.

“The leadership already demonstrated by SfN shows those concerned that a substantial and sustained effort is well underway,” DiCicco-Bloom noted.

The scientific community’s increased focus on reproducibility has resulted in several scientific articles identifying common mistakes and markers of lack of scientific rigor, including issues with:

• Experimental design, such as nonrandom assignment of subjects to groups, lack of blinding in ascertaining results, inadequate power calculations, or lack of appropriate controls
• Data analysis, such as inappropriate statistical analysis or design
• Transparency, such as reporting, publishing, or providing access to specific data, methods, or analyses
• Other factors, such as insufficient reporting of methods or “perverse incentives” in conduct or publishing of science, including rushing publication to meet grant or promotion deadlines

In November, Balice-Gordon and working group co-Chair Oswald Steward of the University of California-Irvine School of Medicine jointly authored a paper in Neuron with recommendations on how to ensure scientific rigor, with a focus on preclinical studies relevant to human neurological and psychiatric disorders. “The review was well-received and was highlighted by speakers in a symposium at the 2014 annual meeting coincident with its publication,” Balice-Gordon said. “We’ve received many emails asking about particulars and thanking us for tackling this topic in a practical, real-world way.”

Drawing on such scientific papers, the Scientific Rigor Working Group, as one of its first tasks, developed Research Practices for Scientific Rigor. These guidelines will serve as a frame for SfN’s programming and activities moving forward. “Time and effort lost to poorly conceived, executed, controlled, or reported work takes a substantial toll on research done in academia and industry,” Balice-Gordon said. “Creating a culture with scientific rigor as a core value is critical.”

Some of the working group’s initiatives consist of direct actions by SfN leadership and committees, whereas others involve lending support for efforts led by other key stakeholders and organizations. In addition to defining research standards, the working group supported relevant programming at Neuroscience 2014, including the Empirical Approaches to Neuroscience and Society Symposium “Improving Animal Models of Neuropsychiatric Disorders” and an NIH-led symposium “Enhancing Reproducibility of Neuroscience Research.” The NIH symposium, co-chaired by Story Landis and Thomas Insel, summarized common causes of poor reproducibility, described actions taken by NIH and journals to improve reliability, offered investigator perspectives, and addressed relevance for training.

For Neuroscience 2015, the Empirical Approaches to Neuroscience and Society Symposium will examine the subject of scientific rigor, and Balice-Gordon has been selected as the symposium chair. “Experimental design, analyses, and statistical treatment of data are central to interpretation and hypothesis testing, and are core to ensuring reproducibility. The symposium at Neuroscience 2015 will be a great way to highlight these issues,” she said. “While I’m still working on the content, I’m hoping one outcome will be identifying best practices and a resource toolbox.”

This year’s annual meeting will also feature a new short course on responsible conduct of research (RCR) viewed through the lens of scientific rigor, for which SfN is consulting with a neuroscientist experienced in developing such training. Charles Mobbs, a professor of neuroscience and geriatrics and palliative medicine at the Mount Sinai School of Medicine, also teaches responsible conduct of research courses at the school. The goal is to provide a short, stand-alone program in which trainees can learn about these issues through a neuroscience-specific lens, a different approach than what is available in many general responsible conduct courses at universities.

The traditional textbook method for teaching courses on responsible conduct of research concentrates mainly on scientific misconduct and philosophical issues surrounding the rationale for ethical behavior, Mobbs said. But in most cases, failure to replicate is not the result of intentional fabrication of findings or willful scientific misconduct, but rather a lax implementation of standard procedures.

“A key challenge in scientific training is that it is one of the few remaining professions taught in the apprenticeship tradition, meaning that students learn to be a scientist by direct instruction under their mentors,” Mobbs said. “This produces a wide variation in acquisition of key skills for best practices in research.”

Thus, Mobbs focuses his classes more on teaching hands-on best practices, which he said also leads to better student engagement. The Neuroscience 2015 short course intends to mirror this method, laying out practical approaches to the problem of reproducibility, such as how to better keep laboratory notebooks and how to write better scientific papers. Like all short courses, the program will be open to all attendees, and the Scientific Rigor Working Group has encouraged neuroscience departments and programs to explore sending trainees for this specialized neuroscience-focused event.

“There’s not only a scientific cost to failing to hold a high bar for rigor and reproducibility, there’s a societal cost that is incalculable,” according to Balice-Gordon. She stressed that while the Society and the field have made “great strides” in raising awareness about the importance of scientific rigor, there is still a long way to go.

People with brain disorders look to scientists to discover new therapies that prevent or reverse their diseases and symptoms. With so many people relying on them, scientists should embrace rigorous standards that help keep research moving forward at a rapid pace and make the most of the field’s intellectual and financial resources, DiCicco-Bloom said.

“It has become clear that our response on scientific rigor must be diverse and systemic, and will include changes in graduate education, scientific methods, journal and grant review procedures, and communication to government and the public,” he added.