Evaluation of the federal railroad administration research and development program

The Transportation Research Board formed the Committee for Evaluation of the Federal Railroad Administration (FRA) Research and Development (R&D) Program at the request of FRA to evaluate the effectiveness of FRA's process for identifying research priorities and the usefulness of FRA R&D products for improving railroad safety The committee evaluated the programs of each of the four Office of R&D divisions (Track and Structures, Rolling Stock, Train Control and Communications, and Human Factors) by examining selected projects (Table S-i, page 2) and reviewed the support functions of planning, evaluation, and management. The committee grouped the questions posed in its charge into three evaluation criteria: Context: Is the R&D activity based on an understanding of industry and FRA needs and priorities, institutional arrangements, and technical resources? Input: Does the R&D activity benefit from communication with relevant parties, including railroad and supplier industries, researchers, the FRA Office of Railroad Safety, and other government agencies? Impact: Are research products of high technical quality do the results of the research find application, and do the applications have demonstrable benefits? The first section below summarizes the committee's conclusions on the performance of the R&D office divisions with respect to these criteria. The second section presents conclusions on support functions. The final section presents recommendations. CONCLUSIONS ON THE R&D DWISIONS' PROGRAMS Context In each of the four divisions, the case studies indicated a clear understanding of rail industry safety concerns and responsiveness to the priorities of FRRs Office of Railroad Safety industry, and workers. No instances of the R&D program overlooking a recognized important source of risk were evident, although the committee did not systematically compare risk sources with R&D resource allocation. Input The FRA R&D divisions are engaging with the Office of Railroad Safety, industry, and researchers and are seeking their input. The case studies included examples of projects developed with input from industry and from the Office of Railroad Safety and cases in which industry demonstrated support through participation in development and testing. The stakeholder review panels formed to support projects in the Human Factors Division have been effective. Such external panels are a promising means of improving communication with industry and other interested parties, especially where research objectives need sharpening in the early stages of projects and where stakeholder engagement is material to success. Some case study projects would have benefited from greater input from industry or the Office of Railroad Safety to clarify objectives or avoid duplication. An apparent low level of industry awareness of some R&D activities limits opportunities for input and indicates a need for more regular and frequent communication between R&D office staff and employees of the railroads, in particular, those responsible for R&D. Evaluations are a vital form of input for program guidance. The Human Factors Division is leading development of evaluation procedures. Impact The case studies demonstrated numerous instances of application of R&D products, including adoption of products by industry and use of research results to support rulemaking. Most current projects reviewed have prospects for producing results that can lead to safety improvements. Overall, the productivity of the R&D program appears good, and the program appears well focused on safety. Some projects have not produced applications. Projects that do not yield tangible results can nevertheless contribute to understanding of safety issues. Such outcomes are worthwhile and are to be expected in a research program. Some programs face obstacles that need to be overcome before faster progress can be achieved. Lack of data hampers progress in human factors research, particularly on fatigue and suicide prevention. The Cab Technology Integration Laboratory (CTIL) suffers from low utilization for reasons related to its location and rules governing its use and for lack of a satisfactory strategic research plan. Implementation of research products depends on actions of the railroads, industry suppliers, the Office of Railroad Safety, and others. Projects such as those supporting positive train control and connected vehicles are components of major industrywide systems and will yield safety benefits only as these systems come into operation. The R&D office can promote implementation through collaboration with these parties and through communication during all stages of its program. CONCLUSIONS ON R&D SUPPORT FUNCTIONS The committee studied four R&D office support functions: communication with industry, the Office of Railroad Safety, researchers, and labor to identify priorities, recruit partners, and implement results; the annual priority-setting process; strategic planning; and program evaluation. The conclusions concentrate on communication; good communication strengthens priority setting, planning, and evaluation. Communication Communication between the R&D office staff and railroad employees responsible for R&D, engineering, and operations is necessary for identifying priorities, recruiting collaborators, and disseminating results. The case study projects showed good examples of collaboration with industry and with railroad professional societies, but in some instances awareness of FRA R&D activities did not appear to be high within the major freight railroads. More regular and frequent communication with the railroads could improve FRA R&D productivity. The FRA Office of Railroad Safety is a primary client of the Office of R&D. FRA staff in both offices described frequent individual meetings to discuss R&D projects linked to rulemaking and periodic briefings by R&D staff for Office of Railroad Safety staff. However, it was not clear to the committee that regular, formal communication occurs between the two offices on all safety areas of mutual interest. In certain research areas, input from parties beyond industry and FRA is needed. In conducting and implementing research on crossing safety and suicide prevention, engagement of state departments of transportation and local law enforcement may be necessary. Communication and collaboration with the FRA regional offices could be helpful in identifying priorities or disseminating results of some projects. The R&D office's goal of increasing the number of R&D projects cofunded by industry is worthwhile. Industry support is evidence that a project is relevant and increases the likelihood of implementation. Cofunding leverages FRA research dollars. Priority Setting The R&D office's formal priority ranking tools are in a trial period during which their utility for supporting decisions is being tested. Their value has yet to be demonstrated, and they do not yet significantly influence decisions. Strategic Planning Development of an R&D strategic plan does not yet appear to be coordinated with established project selection and evaluation processes. Preparation of the strategic plan appears to be an occasional activity rather than an ongoing process, and the office does not track progress toward meeting plan goals. The committee understands that the 2013 R&D strategic plan is seen primarily as a public information source on the R&D program; however, a 5-year strategic plan can be a useful tool in guiding the R&D program and measuring its effectiveness. Evaluation The R&D office is developing an evaluation process that it intends to use routinely. The goal is for each project to be evaluated at certain stages while it is under way and its impact evaluated after completion. The office is beginning to conduct evaluations of selected projects in each division as a learning exercise. Aside from these pilot projects, formal consideration of evaluation needs or methods was not evident in the projects the committee examined. RECOMMENDATIONS Communication 1. To keep aware of industry safety priorities, generate more collaborative projects, and ensure that its products are widely applied, the R&D office should explore opportunities to increase interaction with the engineering, mechanical, operations, and research staffs of the railroads and industry suppliers. 2. The R&D office should ensure arrangements for regular formal and informal communication with Office of Railroad Safety staff in all projects related to Office of Railroad Safety responsibilities. 3. The R&D office should establish external review panels for most projects. The composition and function of the panel would be designed to match the needs of each project. Most panels would include engineering, mechanical, operations, or research employees of the railroads and suppliers as well as independent technical experts. In research on suicide prevention, grade crossing safety, and hazardous materials safety, involvement of state and local highway and police agencies and the FRA regional offices should be considered.