While you are thinking about your 2016 resolutions, consider adopting new practices to counter weaknesses in your analysis. The ideal structured analytic technique to help you get started is the Structured Self-Critique.

Structured Self-Critique is a systematic procedure used to identify weaknesses in analysis, where team members change perspective to become critics rather than supporters of their own analysis. By responding to a list of questions about potential weaknesses in their evidence, assumptions, logic, and cognitive processes, the team is forced to reexamine its own analysis and identify how it might be spectacularly wrong!

How To Get Started

Form a small team composed of the authors, peer reviewers, editors, or other potential stakeholders in the paper. Make sure all the members of the Structured Self-Critique group are wearing the “black hat” of a critic. They should compete among themselves to see who can find the most glaring errors in the analysis. The group should work from a list of known past mistakes, including some or all of the following topics:

• Sources of uncertainty: Identify the sources and types of uncertainty, using these questions:
  • Is the question being analyzed a puzzle or a mystery? Puzzles have answers, and correct answers can be identified if enough pieces of the puzzle are found. A mystery has no single answer; it depends on the future interaction of many factors, known and unknown.
  • How does the team rate the quality and timeliness of its evidence? Are there a greater than usual number of assumptions because of insufficient evidence or the situation’s complexity?
  • Is the team dealing with a relatively stable situation or one that is undergoing, or likely to experience, significant change?
• Analytic process: If the team did not perform the following actions in the initial analysis, consider doing them, or lower the level of confidence in your judgments.
  • Identify alternative hypotheses and seek out information based on these hypotheses.
  • Identify and challenge key assumptions.
  • Seek a broad range of diverse opinions by including analysts from other sectors
• Critical assumptions: If the team has identified key assumptions, focus on those that would have the greatest impact on the analytic judgment, if they turned out to be wrong. How recent and well-documented is the evidence that supports each key assumption? Brainstorm circumstances that could cause each one to be wrong. Would the reversal of any of these assumptions support any alternative hypotheses?
• Diagnostic evidence: IIdentify alternative hypotheses and the most diagnostic items of evidence that enable the team to reject alternative hypotheses. Brainstorm reasonable alternative interpretations of these items of evidence that could make them consistent with alternative hypotheses.
• Information gaps: Reevaluate confidence in your conclusion based on gaps in available information, dated information, and absence of information.
• Missing evidence: Are you missing evidence that one would expect to see in the regular flow of intelligence or open source reporting?
• Anomalous evidence: Is there any item of evidence that would have been important if it had been believed or related to the issue of concern; but was rejected as unimportant because it was not believed or its significance was not known? If so, try to imagine how this item might be a key clue to an emerging alternative hypothesis.
• Changes in the broad environment: Could social, technical, economic, environmental, or political changes play a role in what is happening or will happen? Could these factors have an impact on whether the analysis proves to be right or wrong?
• Alternative decision models: If the analysis deals with decisions by a foreign government or non-state actor, was the group’s judgment about foreign behavior based on a rational actor assumption? If so, consider the potential applicability of other decision models.
• Cultural expertise: If the topic being analyzed involves a foreign or unfamiliar culture, does the team have cultural expertise on thought processes in that culture?
• Deception: Does another country, NGO, or commercial competitor have a motive, opportunity, or means to engage in deception to influence US policy or to change your behavior? Does this country, NGO, or competitor have a history of engaging in denial, deception, or influence operations?

After reviewing these questions, the team must decide a) what additional research is needed, b) what text should be revised, and c) if the level of confidence in the judgments provided is appropriate. If few problems are identified, then the initial judgments have been reaffirmed; if problems emerged from the process, then the paper should not go forward until they have been corrected. More information, such as the best time to use this technique, value added, and potential pitfalls, are found in Structured Analytic Techniques for Intelligence Analysis, 2nd Edition by Richards J. Heuer Jr. and Randolph H. Pherson. Details on this publication can be found here.

Don’t Take Near Misses for Granted

Analytic units will often create their own list of Structured Self-Critique questions that are tailored to their specific work environment. Usually the first items on this list are examples of past errors that the unit does not want to repeat. This is called learning from your mistakes. For most of us, however, it is pretty easy to recall when we made a mistake and remember not to do it again. A bigger challenge is to remember past “near misses” or incidents when we got it wrong-or almost got it wrongand we were lucky that no one noticed.

For example, when a major 5.8 magnitude earthquake hit the US East Coast centered in Mineral, Virginia on August 23, 2011, the North Anna Nuclear Power Plant was only 11 miles away. The earthquake shut down the two nuclear power reactors and three of four diesel generators started up to supply electricity to the safety systems. The initial reaction to this was: “Good news! A nuclear disaster similar to what occurred at Japan’s Fukushima Daiichi plant the previous March has just been averted!” Fortunately a fifth back-up generator was brought on line to replace the broken generator that suffered a coolant leak.

But no one had anticipated that one of the diesel generators would not work. What would have happened if all four or even two or three of the generators failed when the earthquake hit? The plant was designed to survive an earthquake of a magnitude of 5.9 to 6.2 which suggests the event qualified as a very fortunate near miss. The near miss gave Dominion Power the opportunity to review its safety standards and make appropriate adjustments. As analysts we also need to learn from our near misses as well as our failures and use techniques such as the Structured Self-Critique to ensure that we do.