SECTION III
PROGRAM EVALUATION
3.1. INTRODUCTION
An integral and vital aspect of any
fire prevention program is an evaluation of its results. This
section provides some suggestions for measuring and assessing
the effectiveness of programs implemented.
Implementing a program is not the end
of the process. A program must be evaluated continually to ensure
the best use of resources to address a fire problem in the community.
The evaluation process should indicate if a program needs to be
modified or whether its goals and objectives have been achieved.
Achieving these will likely change the character of a community's
fire problems. Therefore, an updated needs analysis may be required
to determine how to maintain the improved level of fire safety
and what priority should be addressed next subject to the availability
of resources.
3.2. PROGRAM EVALUATION METHOD
One method of program evaluation is
explained below. There are a number of components to this evaluation
model. It is important to understand that these components are
interrelated and therefore, all must be applied so that the program
can be implemented easily, monitored continuously, and evaluated
objectively. The first three components address program planning,
the next three address program measurement, and the last three
address evaluating program performance.
Program Planning Components
Goal(s)
A program goal(s) is the intended overall
benefit of the program. It should state what the program is intended
to achieve and should answer an identified need. This would include
the reason(s) why a program was created and consideration of how
the program will evolve in the future. A program goal is linked
to the realization of the overall mission of the department.
The first step to good planning is to
set realistic, achievable goals which reflect the parameters under
which the program must operate. For example, the availability
of fiscal or human resources to apply to a particular activity.
Objectives
Objectives are the measurable outcomes
of a program that can be achieved within a specific period of
time to contribute to the realization of the goal(s). The objectives
should state clearly the quantitative outcomes where appropriate
(e.g. 100 % of apartment buildings will be inspected by a certain
date), as well as qualitative outcomes, such as raised awareness
or attitude shift regarding fire prevention will occur by a certain
date.
In cases where a new initiative has
been launched or where results have not been previously measured
for a program, the first year that the objectives are measured
will be considered baseline information against which all future
results will be compared.
The following are suggested objectives
for programs in the three areas of fire prevention:
Inspections
1) Achieve compliance with fire safety
legislation:
- within a specified time (determined
by follow-up inspections);
- through continued compliance (determined
at subsequent routine inspections);
- through continued compliance for
properties not subject to routine inspection (determined by spot
checks); and
- through voluntary compliance of
properties not subject to inspection.
2) Reduce the potential impact of fire
in properties through:
- the elimination or reduction of
fire hazards to reduce the size and spread of fires;
- containment measures to limit the
size and spread of fires;
- alarm and detection measures to
limit the spread of fire and provide occupants with adequate warning
to respond appropriately;
- suppression measures to limit the
damage caused by fires; and
- fire safety planning information
to increase occupants' knowledge and to limit the damage caused
by fires.
Public Safety Education
1) Increase the fire safety knowledge
and awareness of property occupants:
- to a defined, acceptable level;
- appropriate to the risks for the
targeted occupants.
2) Reduce the incidence and impact
of fires:
- occupants of properties prevent
fire occurrences as a result of fire safe behaviour;
- limit damage and injuries due to
fire by reacting appropriately to fire incidents.
Fire Incident Evaluation
1) Gather sufficient information:
- to allow for proper needs analysis;
- for meaningful evaluation of fire
department programs.
Activities
Activities are actions conducted to
achieve the objectives and goal(s) of a program. A complete description
of the key activities of the program can be developed by asking
questions such as:
- What things need to be done to achieve
a certain result?
- What fiscal, human, and physical
resources are required to achieve these results?
The activities associated with each
type of program are addressed in Section II.
Program Measurement Components
Costs
Both staff and direct operating costs
should be included.
Staff Costs
Salaries, benefits and any other related
costs for staff who carry out the activities of a program. These
costs can be assessed by reviewing the activity reports of staff
and determining the amount of time spent on various activities.
If an activity reporting system is not in place, then a reasonable
estimate of staff time can be used as long as it is clearly identified
as an estimate.
Direct Operating Costs
Transportation, communications, services,
supplies and equipment costs that are required to carry out the
activities of the program.
Success Indicators
Information or data that is tracked
and analyzed to assess the progress of reaching objectives and
goals.
Success indicators should not be developed
in a vacuum as they flow from the objectives and related activities.
If the objectives clearly state how far, how much, by when etc.,
then it will be fairly easy to identify the kind of information
to track and analyze to measure a program's performance.
Program Evaluation Components
Success Assessment
An analysis of a program's progress
towards the achievement of expected outcomes and goals using success
indicators.
A success assessment includes gathering
fire loss information for a community and comparing it to previous
years and to other jurisdictions. Investigating fire scenes will
determine if there is a direct link between a program and improvements
in fire losses. Any statistical and background information used
in the analysis must be objective and stated clearly to minimize
the effects of unrelated factors on the actions of a fire department.
For example, the number of fires per thousand population is an
objective means of measuring the rate of fire occurrence as it
eliminates the impact of changes in population.
Following is an example of a sample
success assessment of the previously suggested objectives for
the three subject areas of fire prevention:
Inspections
- Determine the level of compliance
with fire safety legislation by the:
- incidence of compliance within a
specified time (determined by follow-up inspections);
- incidence of continued compliance
(determined at subsequent routine inspections);
- incidence of continued compliance
by properties not subject to routine inspection (determined by
spot checks);
- incidence of voluntary compliance
by properties not subject to inspection.
- Provide an indication that the fire
safety measures implemented in the inspection program reduced
the impact of fire on evaluated incidents:
- fire hazard elimination and reduction
measures reduced the size and spread of fires;
- containment measures limited the
size and spread of the fires;
- alarm and detection measures limited
the spread of the fire and provided occupants with adequate warning
to respond appropriately;
- suppression measures limited the
damage caused by fires;
- fire safety planning information
received by occupants limited fatalities, injuries and damage
caused by fires.
- Provide an indication that the inspection
program was responsible, in whole or in part, for fire loss reductions.
This could be achieved by thorough a fire incident evaluation
of all fires in properties subject to inspection.
- Determine if fire incidents in the
community may have been prevented as a result of the inspection
program. This could be achieved conducting surveys to seek information
about the fire incidents.
Public Safety Education
- Determine if there was an increase
in fire safety knowledge as a result of the program by surveying
target audiences before and after the implementation of the program.
Establish a starting reference point and compare all subsequent
results to it.
- Determine from the fire incident
evaluations that an increased knowledge of fire safety of occupants
as a result of the program, reduced the impact of evaluated fires.
- Determine if the education program
was responsible, in whole or in part, for fire loss reductions
by conducting a thorough fire incident evaluation of all fires.
- Determine if the fire incidents
in the community may have been prevented as a result of the public
safety education program. This could be achieved by conducting
surveys to seek information about the fire incidents.
Fire Incident Evaluation
Determine whether sufficient information
is being gathered through the fire incident evaluations to allow
for a proper needs analysis and evaluation of the programs.
Why/Why Not Successful
Commentary should be provided on a program's
successes, areas that need improvement and how they could be implemented.
These comments would be based upon, but not limited to, the data
contained in the success indicators and success assessment which
measure a program's level of achievement. Commentary on the objectives,
activities, and program costs may also be helpful as this could
be used to address how resources are being used and how, or why,
resources should be redeployed. Items related to the planning
and execution of activities in future fiscal years should be discussed
as well.
Projected Adverse Impact Without
A Program
This component would highlight the negative
affect on stakeholders and public safety if part or all of the
program ceased to exist. A realistic analysis will provide senior
management with sound information to set priorities.
Conclusion
The on-going evaluation of a department's
fire prevention programs and related activities will provide the
information necessary to plan properly. Meaningful evaluation
results will ensure that a department is achieving its fire safety
objectives and is using available resources effectively and efficiently
to address the community's fire risks. Also, municipal managers
may require the evaluation information when considering whether
to allocate or continue funding for fire department programs.
