From: ESR10-332a - Supplemental Report on Disaster Management

Critics Get Tough on Emergency Preparedness Drills

CALIFORNIA/FLORIDA: An exercise in futility? Emergency preparedness officials these days drill more often than Marine recruits -- hurricanes and terrorism being first and foremost on their agendas -- but a growing chorus of critics is casting doubt on the utility of such drills. Preparedness exercises are supposed to test everyone from first responders to senior level decision-makers, yet long after the tests are complete, critics say, real-life events prove the lessons were not properly learned. "Exercises are not all created equal," said Michael Wermuth, director of homeland security programs at Santa Monica, California-based RAND. "There are a lot of different kinds of exercises, a lot of different methodologies used to conduct exercises. There are exercises that sometimes seem to be destined to ensure success or at least a successful outcome in the exercise." Exercises, the critics say, must be strong enough to expose vulnerabilities -- an outcome many bureaucrats do not want to face.

Ever since the July 2002 National Strategy for Homeland Security instructed the Homeland Security Department to create the National Exercise Program, preparedness programs have proliferated up and down the government ladder. They run the gamut from full-scale live exercises to tabletop drills that ask decision-makers to respond to an escalating series of crises. In one messy affair, first responders in Washington, D.C. simulated a rescue of fake blood-bedaubed passengers from a subway train stopped in a tunnel under the Potomac River. In another, volunteers in Mankato, Minnesota used Halloween favorite Smartie candies to see how quickly drugs could be distributed in a crisis. With enough money and imagination, the possibilities are endless. That, however, does not mean they should be pursued.

Cost remains a major issue. The first TOPOFF drill in 2000 -- which tested responses to WMD attacks -- cost the federal government $3 million. TOPOFF 3, however, cost $21 million after the number of federal agencies involved soared from eighteen to twenty-seven and came to include dozens of state and local agencies and 156 private organizations. As for TOPOFF 4, scheduled for next year, the expense is expected to be even higher, with a precursor event alone -- June's TOPOFF 4 Command Post Exercise - including more than 4,000 people and costing $3.5 million. Not surprisingly, some blame private industry for driving up cost without adding value. "If you're a consultant, it's pretty easy to go anywhere with these templates on [Microsoft] Word and scratch out 'Boise, Idaho,' and put in 'Orlando, Fla.,' " says Eric Noji, retired associate director of the Bioterrorism Preparedness and Response program at the Centers for Disease Control and Prevention and now director of the Pandemic Avian Influenza Preparedness program at the Uniformed Services University.

Problems abound. Participants in tabletop drills may be less than honest, or attempt to brush away difficult situations. During a recent FEMA drill on hurricane preparedness, FEMA seemed unabashed about its lack of preparedness. "It happened a lot - the conversation would stop over something like generators or ice, and a FEMA guy would say, 'Look, don't worry about that, we've got contracts in place, you'll get your million gallons of water a day or whatever,' " recounted one participant. To overcome this problem, independent evaluation is critical to ensure that exercises accurately reflect capabilities and deficiencies.

"All these exercises don't mean anything unless there is some type of after-action report, [but] some people in some agencies see the exercise as the end in itself rather than a means to an end," said professor Carl Osaki of the University of Washington, who has designed several simulations. "A lot of times the findings of the after-action reports require additional training or policy. Sometimes [producing the reports is] time-consuming, or they're costly. So once they hit some of those barriers, the after-action report is sometimes seen as an academic exercise."

-- Source: Zack Phillips' GovExec report, which can be found at: http://www.govexec.com/story_page.cfm?articleid=35549&dcn=todaysnews

ERRI General Analysis of Disaster Drills
By C. L. Staten, ERRI Senior National Security Analyst

CHICAGO, IL: One of the primary problems with disaster drills is that it is often difficult to place the first responders into "actual crisis mode" during a simulation (drill). Many, if not all, veteran first responders are under the impression that they already have more than enough work to do, without attending some sort of "#$@^@&*%" (expletive deleted) disaster drill. And, particularly in busy urban systems -- at least to some extent --that may be true. But, that attitude also doesn't really take into account the fact that a terrorist event may be tactically very different than a bus crash.

And, maybe more important than that is the fact that few "old-time" emergency responders actually take disaster drills very seriously...mainly for at least three reasons:

• <1> Few Chiefs/Administrators/Supervisors take the drills earnestly and they usually just "go through the motions" in order to meet some requirement...be it from FEMA, DHS, or one or more of the local hospitals...who are mandated to participate in a disaster drill every so often in order to maintain their accreditation. (1)
• <2> Many things in a disaster drill are frequently "simulated" to facilitate the smooth flow of the exercise. The problem, as is pointed out in the accompanying article, is that it is far more difficult to provide the supplies, obtain the necessary manpower, or carry out a set of complicated procedures in real life...than it is to just say so during a drill.
• <3> Maybe most important to actually meeting the real goals of a disaster drill is the fact, that in many emergency systems, it is virtually impossible to fail a disaster exercise. One way or the other...inadequacies or overt mistakes will not be documented, will be purposely overlooked, will suddenly become "notational" or "simulated," (2) or simply ignored by the leadership of the organization running the drill and/or carrying out an evaluation thereof.

This commonly occurs for at least four reasons:

• A. Fear of Failure...no Chief/Administrator wants to admit that their leadership resulted in failure.
• B. Lack of adequate planning, resources, knowledge, and/or management skills
• C. After all...it is only a drill, and mistakes don't count. The prevailing attitude is "don't worry about it."
• D. If the drill is being observed by the press and/or politicians from the jurisdiction having authority over the participants...mistakes, oversights, omissions, or ANY other problems are guaranteed to be disavowed or "glossed over," if that is at all possible. (Again, please see #3A)

And, finally, a major problem in the management and critique of disaster drills is that they are normally evaluated within their own system...by their own intra-department personnel. These "evaluator personnel," who are dependant on their Chief/Administrator for future promotions and other benefits, are smart enough to read #3A above and insure that their evaluation does not reflect that THEIR CHIEF'S "leadership resulted in failure." In other words, the evaluation process is incestuous and if needed the evaluators will "cook the books" so as to not reflect discredit on their Chief or department.(3)

SOME CONCLUSIONS/RECOMMENDATIONS

Instead of the circumstances outlined earlier in this analysis, it is imperative that the following things take place in order to have an effective disaster drill:

• <1.> All disaster drills are supposed to be "learning experiences" and mistakes are supposed to happen. That is how the effective manager can learn about problems within the system or the plan itself. Criticism of the drill must be honest and given in a constructive manner in order to be useful. It should be clearly understood at the onset of planning for any exercise that no repercussions, either positive or negative, will be taken against anyone as the result of a training exercise.
• <2.> Leaders should realize that it is generally counter-productive to "self-evaluate," except during normal training and early/preliminary learning experiences. Truly effective evaluation a major disaster exercise should utilize unbiased and knowledgeable outside evaluators. Bottom-line; exercise evaluators should NOT have ANY vested interest in the success or failure of the exercise. Then, it is possible for fairness and honesty in the evaluation to prevail.
• <3.> The use of "simulation" or "notational action" (see #2 in references below) during a drill, should be examined critically and disallowed if they do not exist as an "option" in the disaster/exercise plan that is being tested and evaluated. While great latitude must be given to an emergency manager/incident commander (and their staff) at the scene, it is also necessary to assess and document issues/difficulties that are caused by a faulty plan or procedure.
• <4.> We agree with whole-heartedly with Kerry Fosher, of the New England Center for Emergency Preparedness, that a great deal is learned about a disaster plan the day before the exercise takes place. The keys to an effective exercise, or the management of a real incident, include; a common understanding of expectations, effective communication channels, mutual respect and cooperation, and the creation and implementation of a well understood interagency plan.
• <5> It is common knowledge that every "incident commander," of any repute, will do their best to alter events to save as many (simulated) lives as possible. But, an important question must then arise: why allow/rationalize/ institutionalize "command free-lancing" to resolve mistakes that exist in an official plan? If it is shown to be necessary for the incident commander to constantly "improvise" to achieve a successful outcome during a drill...change the plan.

Finally, any disaster drill or exercise must be part of a larger and cyclical planning and revision process. Try this series of actions to develop an effective plan and response to a major disaster:

• A. Write a plan
• B. Train all participants for their part in the plan
• C. Conduct a drill to test the plan and the personnel
• D. Evaluate the effectiveness of the plan
• E. Revise the plan as necessary to achieve the desired end-outcome objectives
• F. Repeat as necessary...

References:

(1) Joint Commission on Accreditation of Healthcare Organizations (JCAHO), http://www.jcrinc.com/index.asp?durki=2

(2) Notational, in this case, means that "we write down that we did it...whether it happened or not." Yeah, that's right...Yeah, we did it. that's the ticket...

(3) In fact, in some emergency service departments, it is actually a formal rules and regulations violation to "reflect discredit" on their leaders or department. Such violations can even be punished by disciplinary action, should the Chief/Administrator decide to do so.

[Editor's note/About the Author: This commentary was written by a veteran emergency response manager, who has managed many disaster drills...including a series of major drills that were openly conducted in front of an audience of emergency responders at the National Association of EMT's annual meetings. Additionally, he wrote, taught, tested, and revised the official EMS mass casualty plan for the Chicago Fire Department. And, he has acted as an observer and advisor to local emergency systems at several nationally/internationally prominent drills and actual disaster incidents.]