Abstracts

The role of the RAC in the peer review of CVM’s risk assessments
Barry Hooberman, FDA/CVM

Risk assessment activity at the Center for Veterinary Medicine (CVM) is characterized by a number of factors that result in a set of distinct needs for peer review compared to those at the Center for Food Safety and Analysis, or at the Food Safety and Inspection Service. Organizationally, CVM is limited in size and resources, which restricts the opportunities for internal peer review. Functionally, most of the risk assessment activity at CVM is focused on new or past drug approvals, which can present unique data confidentiality and regulatory issues in performing and reviewing risk assessments. From a subject matter viewpoint, much of the risk assessment activity at CVM relates to antimicrobial resistance in foodborne pathogens, which has important distinctions from microbial risk assessments.

CVM has employed a number of methods for peer review of its risk assessments, including open public meetings, solicitation of public comments in docket form, and the use of outside experts in the planning, execution, and review of its risk assessments. In particular, CVM has successfully used members of the Interagency Risk Assessment Consortium (RAC) as an important and valuable resource in conducting its risk assessments.

This presentation will discuss CVM’s “lessons learned” in its risk assessment activities, the importance of peer review during and after the risk assessment, and the importance of the RAC as a mechanism for peer review.

USDA/FSIS’s needs for Peer Review of Risk Assessments

Carol Maczka, USDA/FSIS

Peer review is an integral part of the risk assessment process at FSIS. In part, this review will be provided by the public during comment periods in response to announcements and public meetings; however, the agency believes it is important that scientific review also be sought. For projects with less economic impact, peer review may consists solely of internal reviews or review by knowledgeable experts in sister agencies. For large assessments that will be used to inform regulations that have a significant economic impact, independent external expert review is sought.

For the risk assessments conducted to date, the Agency has used a range of peer review mechanisms including review by inter- and intra-agency specialists; the National Advisory Committee on Microbiological Criteria for Foods; the Academy of Sciences; contractual arrangements and IPAs with subject area experts, and the Risk Assessment Consortium. Each has its advantages and disadvantages.

Based on its experience, the Agency has come to recognize the need for a peer-review system that is able to cost-effectively access a broad range of scientific expertise in a timely manner. Requirements for peer reviewers and elements of a good peer-review process will be discussed. Challenges facing the Agency with respect to future reviews will also be presented.

Developing EPA’s Peer Review Program
Dorothy E. Patton, Consultant

Peer review has a long history at the U.S. Environmental Protection Agency and takes several different forms, some modeled after customary academic practice and some following statutory mandates. At the same time, EPA’s peer review practices have at times been uneven, varying in scope and quality. Beginning in 1992, responding to recommendations from the EPA Science Advisory Board (and, later, the General Accounting Office), EPA embarked on an agency-wide program to expand and strengthen its peer review policies and practices. The expansion process presented challenges on several different fronts and, after numerous stops and starts, led to a range of institutional changes. Today, agencies responding to the proposed OMB guidelines may face similar challenges. These agencies may benefit from learning about one agency’s approach to instituting more rigorous peer review practices.

Introduction: What is peer review?
Heather Douglas

In this talk, I will provide general definitions for what is peer review, including differentiations with public comment (which may not include “peers”) and peer consultations (which are generally less formal than peer “review”). In addition, I will describe differences among journal peer review, funding peer review, and regulatory peer review, in terms of both the goals of the review and the processes used in the review. These distinctions should help lay the groundwork for the day’s discussions.

Strengths and Limitations of Regulatory Peer Review
Heather Douglas

Regulatory peer review can provide a useful check on the methods used to develop science-based regulatory documents. In particular, it is a good way to ensure that erroneous assumptions or faulty data are not relied upon in those documents. However, depending on the complexity of the document and the scope of the charge given to peer reviewers, the peer review process will require great care in the implementation of the peer review process, from the selection of reviewers to the drafting of the final review report. In this talk, I will describe a spectrum of expectations for the peer review process. As potential expectations increase, so do the difficulties and fundamental dilemmas for the peer review process. This talk will describe those dilemmas (e.g. the desire to avoid conflicts of interest vs. the need for a range of expertise, or the needs for balance in peer review panels vs. the desire for consensus in the outcome) as well as why they are unavoidable if one wants a great deal of assurance from regulatory peer review.

Evolution of EPA’s peer review program
Jeff Morris, EPA/ORD

Using Dorothy Patton's presentation “Developing EPA’s peer review program” as a point of departure, this talk discusses EPA's progress over the past 2-3 years in enhancing the use of peer review for scientific and technical work products that inform Agency decision making. Specifically, this talk will cover: the increase in numbers of peer reviews at EPA, peer review as a cornerstone of EPA's Information Quality Guidelines, peer review in the Agency's decision process, and OMB's proposed peer review standards for regulatory science.

Federal Review of Risk Assessments
James D. Schaub, USDA/ORACBA

Federal regulatory actions are expected to be based on sound scientific and economic analysis. Public policy and the public’s expectations have trended toward more and better analyses with more accountability, openness, and accessibility to these analyses and underlying data. Legislation, Executive Orders, Administration guidance documents, and agency policies communicate this. Agencies have established review and clearance systems for risk assessments. These systems do not ensure the highest standards for peer review. Nor does the release of draft risk assessments to the public ensure adequate peer review. Although the Office of Information and Regulatory Affairs of the Office of Management and Budget is the ultimate quality check for regulatory analyses, there is a strong case for agency responsibility for good peer review. USDA’s experience with review of risk assessment illustrates strengths and weaknesses of federal peer review.

Peer Review: Challenges Raised by OMB’s Guidelines.
Leslie Hushka, Ph.D., ExxonMobil Biomedical Sciences, Inc.

Over the last few years there has been considerable discussion in scientific, policy and public fora on the effectiveness of peer review systems, including those employed by journals, research institutes or agencies. These discussions have focused on how to select qualified reviewers, develop appropriate instructions for reviewers, select a suitable format for the review, and more importantly, how to ensure that peer reviews are scientifically rigorous and appropriate for the end use of the information. In August 2003, OMB released a draft bulletin that updates it's Information Quality Guidelines (IQG) to include additional guidance on how federal agencies should peer review scientific, technical, and economic information that will serve as the basis for a 'significant regulatory decision'. This draft guidance reinforces these earlier discussions and raises many new questions. It also identifies a new topic for debate, briefly "Should peer review for 'significant' information used to support a regulatory decision be different from other agency peer reviews, and if so, how?" This presentation will highlight these new questions and challenges posed by OMB's draft guidelines in an effort to assist agencies and other groups as they prepare to respond to OMB and modify their existing practices.

CFSAN’s Peer Review Process for Risk Assessments
Robert L. Buchanan, Sherri Dennis, and Marianne Miliotis
FDA/CFSAN

One of the foundations of rigorous scientific programs is rigorous critical reviews of scientific reports and analyses. This is particularly important for risk assessments where the need to make assumptions and to make decisions about the suitability of data has the potential for unintentional biases.

CFSAN’s current process includes the following activities:

  • Announce intention in the Federal Register and request submission of data.
  • Hold public meetings to describe the key assumptions, data, and modeling approaches. This is likely to include submission of the materials to one or more FDA advisory committees.
  • Seek input from stakeholders, scientific experts and advisory committees.
  • Conduct an initial extensive review of report/ model by government employees, including special governments employees who serve in an advisory capacity.
  • Issue draft of the risk assessments to the public for review.
  • Issue revised final risk assessment.

CFSAN’s intention is to formalize this peer review process to assure that CFSAN’s risk assessments meet all of the requirement of FDA’s newly adopted principles for the conduct of risk assessments that support regulatory programs. These reviews will focus on improving both the science underlying these complex risk assessments and the acceptance of the risk assessments by our stakeholders. Requirements for this process are likely to include: (1) a description of the process (activities) by which the peer review is to be conducted; (2) a description of the qualification of reviewers; (3) the specific elements that a reviewer should consider in evaluating food safety risk assessment; (4) an evaluation of benefits/ limitations of the peer review elements, and (5) pragmatic constraints that may limit the alternatives recommended by peer reviewers.

The Economic Benefits of Economic Peer Review
Richard Williams, FDA/CFSAN

There are both benefits and costs associated with peer reviewing economic analysis (as with any analysis). Benefits depend on 1) the expertise of the agency economists generating the study as well as the oversight of those studies; 2) the types and qualities of expertise of the peer reviewers; 3) the transparency of the peer review process; 4) whether the errors are likely to be corrected in the analysis ; 5) the societal value of the decisions that are analyzed; and, 6) how much the risk management decision depends on the economic analysis. Any set of rules considered for when a peer review is worthwhile should consider these factors in a benefit/cost analysis.