To support policy and resource allocation decisions, the Department of Homeland Security (DHS) and the Transportation Security Administration (TSA) have developed a suite of tools and processes for conducting risk assessments. One such tool is the Risk Management Analysis Tool (RMAT) developed by the Boeing Company and TSA in consultation with private sector and governmental members of a risk management working group. In December 2010, TSA asked RAND to evaluate whether RMAT provides results that are valid for TSA’s risk-assessment needs. This report describes RAND’s approach to this assessment and our findings.
RMAT simulates terrorist behavior and success in attacking vulnerabilities in the domestic commercial air transportation system. In doing so, it draws on estimates of terrorist resources, capabilities, preferences, decision processes, intelligence collection, and operational planning. It describes how the layers of security protecting the air transportation system are likely to perform in the face of a range of more than 60 types of attack. It draws on detailed blast and other physical modeling to understand the damage produced by different weapons and attacks and calculates the direct and indirect economic consequences of that damage. As such, the tool is designed to provide vital information that can help TSA understand the risks to which the entire air transportation system is exposed and develop ways to improve it.
RAND’s approach to validating RMAT required first establishing TSA’s intended uses and requirements for risk assessment and then evaluating which of those requirements RMAT can satisfy. We have not validated RMAT against a set of original requirements for the system, because RMAT has evolved over time without such a guiding set of requirements. Instead, therefore, we evaluate TSA’s current and broad risk-assessment requirements that RMAT can validly support.
We divided the validation effort into four substantive research questions:
1. Are the adversary behavior and air transportation system conceptual models valid, and are the data used to support them adequate? (Chapters Two and Three)
2. Are the sources and methods for populating the RMAT model with data sufficient to ensure their validity? (Chapter Four)
3. Does the RMAT code, as implemented, perform in the way it was designed to? (Chapter Five)
4. Can risk estimates from RMAT be used in the ways TSA intends? (Chapter Six)
These validation efforts considered diverse sources of evidence, including published scientific literature, elicited judgments from subject matter experts, considerations of logic and reasonableness, historical evidence, and quantitative empirical analysis of RMAT and its outputs. Each of these chapters includes detailed observations about RMAT strengths and weaknesses and concludes with a set of recommendations for further developing RMAT. In this Summary, we highlight only the main findings from the study.
About the RAND Corporation
“The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND focuses on the issues that matter most such as health, education, national security, international affairs, law and business, the environment, and more. With a research staff consisting of some of the world’s preeminent minds, RAND has been expanding the boundaries of human knowledge for more than 60 years. As a nonpartisan organization, RAND is widely respected for operating independent of political and commercial pressures.”