The Environmental Protection Agency (EPA) is required to use methods “consistent with the best available science” to evaluate scientific evidence on chemical health risks. These methods are critical because they shape EPA’s decision-making on chemicals evaluated under the Toxic Substances Control Act (TSCA), which has profound implications for public health.
In 2018, EPA released its systematic review method – the method for evaluating scientific evidence – for the first 10 chemical risk evaluations under amended TSCA. The 2018 TSCA systematic review method raised major concerns for scientists as it was not “consistent with the best available science.” Reasons for this included the method:
- lacked a proper protocol for each assessment,
- failed to appropriately evaluate study quality, and
- could improperly exclude valid evidence from consideration.
This culminated in a National Academies of Sciences, Engineering and Medicine (NASEM) report recommending EPA stop using this method (Figure 1).
In February 2021, EPA said it would drop the method. While EPA said they did, they replaced it with something equally problematic that retains many of the flaws of the original method.
EPA’s new method, the Draft Toxic Substances Control Act (TSCA) Systematic Review Protocol (2021 Draft TSCA Method), failed to address key recommendations from the NASEM.
As we have highlighted, the 2021 Draft TSCA Method is merely a rebrand of EPA’s 2018 method, again falling short on protocol development, study inclusion criteria, study quality assessment, and evidence integration (Figure 2).
(1) Protocol Development
A pre-published protocol is critical for completing a transparent and unbiased review. The NASEM recommended:
A systematic review protocol that details the pre-specified methods, including eligibility and critical appraisal criteria, and that is peer-reviewed and publicly posted before the review commences should be prepared.
While the 2021 Draft TSCA Method states EPA followed this recommendation and the TSCA requirement to publicly release a protocol before starting any risk evaluation, this is false: EPA was already conducting chemical risk evaluations while drafting this method. This is neither transparent nor representative of the “best available science.”
(2) Study Inclusion Criteria
A PECO (Population, Exposure, Comparator, Outcome) statement provides criteria that researchers use to decide which studies to include or exclude in a systematic review. The NASEM recommended:
Eligibility criteria need to be based on PECO statements that are formulated in a standard way and need to be predefined in the protocol.
EPA’s PECO statements do not provide a clear, consistent, reasonable, or appropriate basis for determining which studies are relevant for hazard assessment. For example, EPA uses PECO statements that emphasize organ-level effects and observable effects like birth defects and cancer, excluding other important endpoints like hormone alterations and cellular changes, which could dramatically decrease the health effects EPA considers.
(3) Study Quality Assessment
Study quality refers to assessing if researchers conducted their study to the highest possible standard. The NASEM recommended:
Use established tools for assessing risk of bias and study quality such as those developed for use by OHAT or the Navigation Guide, or, at a minimum, remove inappropriate appraisal criteria from the current tools.
EPA claims it followed these recommendations in its 2021 Draft TSCA Method, but its method is incompatible with validated best practice methods and includes inappropriate appraisal criteria, such as statistical power.
NASEM also recommended:
Do not use numeric scores to evaluate studies.
While the 2021 Draft TSCA Method states the Agency followed this recommendation, this is false. EPA still uses a numeric scoring system for studies – the same approach that the NASEM criticized.
EPA also ignored NASEM’s recommendation:
Do not exclude studies based on risk of bias, study quality, or reporting quality.
Contrary to NASEM’s advice, EPA decided it can exclude a study for being “critically deficient” in just one area, instead of looking at the study holistically.
(4) Evidence Integration
Evidence integration refers to the overall interpretation of various streams of evidence (animal, human, in vitro) to develop findings and conclusions. The NASEM recommended:
Separate evidence synthesis from evidence integration.
Here again, the 2021 Draft TSCA Method falls short, failing to draw a clear distinction between evidence synthesis (looking at human, animal, in vitro evidence separately) and evidence integration (looking at those streams together). EPA does not use the term “evidence synthesis” at all, instead creating confusion by first describing “integration” as evaluating separate bodies of human, animal, and in vitro evidence, then later describing it as merging these bodies of evidence together.
Why this matters
By failing to comply with critical steps for a systematic review, EPA’s 2021 Draft TSCA Method will result in biased reviews of the evidence. This is a direct threat to EPA’s mission to protect human and environmental health.
There is a solution. EPA can follow recommendations from the NASEM and adopt best practices for conducting systematic reviews in environmental health like the Navigation Guide, the National Toxicology Program’s (NTP) Office of Health Assessment and Translation (OHAT) method, and EPA’s Integrated Risk Information System (IRIS) method (if NASEM recommendations for the IRIS method are incorporated).
For more on this, see our study, public comments, and blog.
About the author
Courtney Cooper, MPH is a Science Associate for the Science & Policy team, working to promote evidence-based decision making and best practices for systematic reviews in environmental health. She graduated with a BS in Public Health and Minor in Dance from the University of South Carolina and received an MPH from the University of San Francisco. She is interested in advancing maternal/reproductive health and environmental health.
You must be logged in to post a comment.