Raytheon Studies the Decision-Making Process of Intel Analysts

Partnering with Penn State University and intelligence analysts, Raytheon is conducting cognitive analytic-gaming exercises, called the User-Centric Grand Challenge, to learn more about how intelligence analysts make decisions. The goal is to provide richer analytics approaches to help analysts make more informed decisions and accurate intelligence products.

Raytheon data scientists and engineers use instrumented software tools to assess the analytic process and to recommend alternate approaches. This process – backed with scientific rigor – gains quality improvements and helps to find efficiencies for intelligence and defense customers facing a deluge of multi-INT data.

Using a gaming concept along with Raytheon data and analytic software to reproduce a day-in-the-life in an intelligence cell, the cognitive researchers help analyst teams learn what they can do to improve their day-to-day productivity and product quality by closely studying how they make decisions.

"One way to think of this is that we are analyzing the analysts," said Karen Ebling, Raytheon Intelligence and Information Systems analytics strategy director. "We're conducting deep scientific study to ultimately help analysts create the best intel products possible."

Penn State serves as the industry's academic research partner by contributing high fidelity yet fictitious data sets to support the User-Centric Grand Challenge.

"Over a two-year period, Penn State created an unclassified, physical and soft data sets based on real information from the insurgency in Baghdad," said Dr. David Hall, Dean of Penn State's College of Information Sciences and Technology. While many industry and academic research partnerships exist, there has not been priority demand for research advancing the cognitive science of the analytic tradecraft.
"Penn State offered the data sets to Raytheon since we already have a strong working relationship, " Hall said.

The focus of the data sets, called synthetic counter-insurgency (SYNCOIN), is to support the development and implementation of fusion and analysis processes, data stores design, and the development of process flows and interfaces required for both automated processing and user in-the-loop analysis to enhance situation awareness and understanding.

"SYNCOIN was originally developed under a multi-year Multidisciplinary University Research Initiative sponsored by the Army Research Office," said Jake Graham, the creator of the SYNCOIN data and who is a professor of practice at Penn State, director of the Center for Network-Centric Cognition and Information Fusion and a retired Marine Corps colonel.

Research partnerships with academia, government and industry foster innovation test-beds to prove the functionality of a solution.
"By including intelligence analysts, Raytheon gains insight into the analytic tradecraft. At the same time, we give the analysts improved ways to approach problem solving," said Ebling.

Overcoming Intel Analysts Biases to Help Prevent National Security Lapses
Raytheon is also studying common judgment biases that unknowingly crop up while analysts perform their daily tasks.

When analyzing all forms of data, analysts subconsciously apply decision heuristics, or mental shortcuts intended to reduce complex problems to simpler ones. While these heuristics ease the "cognitive load" of the analyst and often produce accurate judgments, they sometimes result in judgment biases, or errors caused by incomplete mental processing. The negative effects of judgment biases can be reduced through focused corrective measures known as analytic multipliers, designed to improve the analytic quality and accuracy.

The objective of understanding analysts' biases through analytic decision games is to prevent intelligence analysis errors.

"By countering analysts' judgment and intellective biases, we can increase analytic potential and improve analysts' estimates and conclusions by avoiding common mental pitfalls. With greater analytic yield, the intelligence process becomes more reliable and efficient to prevent future national security lapses," said Don Kretz, Raytheon lead cognitive scientist. In 2001 and 2004, the U.S. government issued reports examining the Intelligence Community's analytic practices and Congress passed the Intelligence Reform and Terrorism Prevention Act of 2004 to reform the intelligence community and intelligence-related activities, in particular mandating the community to adopt the use of "alternative analyses."

"This means that intelligence community analytic organizations need to institutionalize sustained, collaborative efforts by analysts to question their judgments and underlying assumptions, employing both critical and creative modes of thought," said Kretz, a former military and civilian intelligence analyst. "For this approach to be effective, significant changes in the cultures and business processes of analytic organizations will be required."

Raytheon's User-Centric Analytics Grand Challenge
The User-Centric Grand Challenge improves an analyst's efficiency by overcoming biases applying time-saving devices to give analysts more time to think. Raytheon based the corrective measures, or analytic multipliers, on a pilot game with Raytheon data scientists and engineers. Raytheon instrumented the software tools to assess the analyst's thought and decision-making patterns which were reviewed by the research team.
Raytheon evaluated the results to develop an alternate and more efficient approach by displacing the judgment biases with analytic multipliers. The next phase of the research will move forward this fall by testing the analytic multipliers on intelligence analysts currently working in the GEOINT field.

More Research and Development
Kretz said it would be helpful for the government to fund more cognitive science research, in the way that industry and academia are partnering.

"More R&D funds should be committed toward the cognitive science of analysis by teaming with industry like Raytheon in the grand challenge," Kretz said.

Through R&D efforts like the User-Centric Analytics Grand Challenge, Raytheon hopes it will trigger interest and invest in cognitive studies that will better serve government customers.