EPA Human Studies Review Board Peer Review Shows that Not All Peer Reviews Are Equal: A Tale of Two Peer Reviews On Formaldehyde

As described in our blog post entitled The EPA Human Studies Review Board Scientific Review Highlights that Major Revisions Are Needed to EPA’s Draft IRIS Formaldehyde Assessment, two important reviews have been released evaluating EPA’s formaldehyde science. On August 9, the National Academies of Sciences, Engineering, and Medicine Committee on the Review of EPA’s 2022 Draft Formaldehyde Assessment (NASEM committee) released a report describing its review of the adequacy and transparency of EPA’s methods in the 2022 Draft IRIS Formaldehyde Assessment. On August 23, the EPA Human Studies Review Board (HSRB) approved a report that described its review of the ethics and science related to four studies that EPA used in a weight-of-evidence evaluation for acute sensory irritation resulting from formaldehyde exposure. 

This post provides a comparison of panel development and the review processes used by the NASEM committee and the HSRB. 

The Importance of the Charge Questions

As described previously, the NASEM committee did not conduct an independent assessment of the formaldehyde science. Instead, it evaluated whether the 2022 Draft IRIS Formaldehyde Assessment “adequately and transparently” evaluated the scientific literature and used appropriate methods to synthesize the state of the science. The NASEM committee was not charged with commenting on the full body of literature relevant to the hazards and risks of formaldehyde, nor was it charged with reviewing alternative scientific opinions. 

In contrast, the HSRB was charged with reviewing EPA’s weight of the evidence evaluation of the science related to formaldehyde induced sensory irritation as presented in a paper provided by the EPA Office of Chemical Safety and Pollution Prevention (OCSPP). The HSRB conducted a deep dive into the literature and considered material provided by EPA, public commenters, and publicly available information including research articles. While the NASEM committee considered EPA’s transparency and consistency with guidance, the HSRB conducted an in-depth scientific evaluation. 

The different charge questions also led to differences in the standard each committee used. The NASEM committee repeatedly referred to evaluating whether EPA’s approach was consistent with “its state of practice” methods. While this term was not defined, the NASEM committee focused on whether EPA’s approaches were consistent with EPA guidance documents. In comparison, the HSRB recognized the Toxic Substances Control Act (TSCA) scientific standard that requires that decisions be based on the “best available science and on the weight of evidence.”

The Important Role of FACA and Good Government Approaches in Ensuring Quality Reviews

The Federal Advisory Committee Act (FACA) governs the operation of federal advisory committees and emphasizes public involvement through open meetings and reporting. FACA ensures transparency with the public in committee operations and requires substantive engagement with the public. It also requires specific transparency, balance, and independence requirements in the peer review process. Both the NASEM committee and the HSRB were FACA committees; however, as described below and as summarized in Table 1, there were significant differences in their approaches to ensuring consistency with FACA requirements.

Diversity and Balance

Section 5(b)(2) of the FACA requires “…the membership of the advisory committee to be fairly balanced in terms of the points of view represented and the functions to be performed by the advisory committee.” In discussing the perspectives required, FACA regulations require the consideration of those with divergent points of view as well as perspectives including “those of consumers, technical experts, the public at-large, academia, business, or other sector.” The FACA regulations require the agencies to have a plan for obtaining balanced membership, and the HSRB has a balance plan. As is summarized in Table 1, and consistent with the requirements of the HSRB Bylaws and balance plan, the HSRB panel included experts from four sectors: state government, federal government, academia, and consulting. The HSRB experts included representatives from 11 states.

NASEM also has a policy on composition and balance that requires the consideration of expertise, perspectives, objectivity, and diversity. In addition, NASEM must also follow FACA provisions consistent with those described above for the HSRB. Despite these policies and requirements, as is shown in Table 1, the NASEM committee was significantly less diverse in its perspectives and in geographic representation, including only experts from academia and state government. These experts included representatives from only 8 states.

EPA’s Peer Review Handbook discusses the importance of avoiding the use of the same reviewers on multiple assessments and suggests using different individuals each time a product is sent back to peer review to provide a broader perspective. As shown in the table below, 6 of the 13 NASEM reviewers had previous involvement in reviewing EPA’s science on formaldehyde. ¹

Objectivity and Bias

The NASEM policy on composition and balance includes consideration of objectivity, including whether “an individual may have strongly held views or biases, or may be closely associated with a group that has taken a strong position, on an issue before the committee.” While the HSRB panel included experts that had not previously reviewed and commented on EPA formaldehyde assessments, the NASEM committee included 5 members that were on a previous NASEM committee in 2011 that reviewed EPA’s previous formaldehyde draft assessment. Another NASEM committee member was also a reviewer of the previous formaldehyde draft assessment. In addition, the NASEM staff lead had previously worked at EPA and had provided internal input on the draft assessment. These relationships, including public comments on a previous draft formaldehyde assessment, may have compromised the objectivity of the NASEM review.

Transparency

FACA also requires that meetings are open to the public and that records of the committee are available for public inspection and copying. The HSRB charter cites to FACA requirements, National Archives requirements, and FOIA requirements. Accordingly, the HSRB conducted all its full committee work in the public eye, conducting public meetings and, consistent with these requirements, sharing all draft reports with the public. In stark contrast, the NASEM committee only conducted some of its work in public, had full committee meetings that were not open to the public, and did not share any draft reports publicly. The NASEM committee approach is not consistent with FACA requirements as implemented by EPA.

Public Engagement

FACA requires that the public is afforded an opportunity to provide input into a process that may form the basis of government decisions. While both the NASEM committee and HSRB had comment periods to hear from the public, only HSRB truly engaged with the public and considered this input. The NASEM committee report, beyond mentioning that they had public comment periods, makes no mention of whether or how public input was used to inform its report. Considering the limited nature of the NASEM committee charge, and that the NASEM report states that the committee “was not charged with…. reviewing alternative opinions of EPA’s assessment,” it is doubtful that public comments were considered relevant to the NASEM committee. In contrast, the HSRB engaged in scientific discussion with commenters and referenced their input in the final HSRB report. While the HSRB approach was consistent with the goals of FACA, the NASEM review was not.

Level of Rigor

As described above, the NASEM charge was very limited. The committee only evaluated consistency with guidelines and alignment with previous recommendations relating to general risk assessment methodological approaches (the “state of practice”). When discussing scientific issues, including hazard and dose-response which are key elements of the draft formaldehyde assessment, the NASEM review was constrained. Alternatively, as discussed here, HSRB dove deeply into the underlying data in each study while looking closely at methods, statistics, author conclusions, and EPA’s conclusions. HSRB conducted its own evaluation to inform EPA’s weight-of-evidence evaluation.

Surprisingly, despite its limited review, the NASEM committee report, while noting instances where EPA did not follow its own guidelines, does not make recommendations related to some of the key failures in EPA’s assessment. For instance, the NASEM report notes that terms used in EPA’s strength-of-evidence framework, which is a key element in EPA’s approach, are not consistent with terms used inside or outside of EPA and are used “inappropriately.” While the NASEM report states that “[t]he use of these terms represents an unnecessary source of inconsistency with the state of practice,” the NASEM report does not provide any recommendations to EPA to address this inconsistency. Thus, despite the limited scope of the charge, the review does not appear to be as rigorous as a detailed reading of the NASEM committee report seems to support.

The NASEM and HSRB Comparison Shows that a More Robust FACA Review Leads to a More Overtly Critical Peer Review Report

FACA is designed to provide the federal government with independent, unbiased input. The finding that the more rigorous review led to a more critical review of the EPA work product is actually good news. In this case the HSRB review, which was more consistent with FACA requirements when compared to the NASEM committee review, provided a report that was more critical of EPA’s scientific analyses. 

FACA requirements are precise and rigorous for a reason. In 1972, when FACA was enacted, Congress recognized the importance of providing the government with expert objective advice. This objective input strengthens government science and therefore leads to improved regulatory decision making. FACA requirements improve peer reviews and should be the gold standard, including within EPA and NASEM.

Table 1. Overview of NASEM Committee and EPA HSRB Consistency with FACA

¹ In addition, as discussed in comments sent to NASEM in 2022, ACC’s analysis showed that the 13 NASEM panel members have served on over 220 federal advisory committees.