Evaluation: Proving Risk Watch Works

Evaluation is a vital part of a successful program and must be a priority for your coalition. Simply put, evaluation is the measurement of change. While change can be measured in many different ways, the three areas that are generally measured for school-based programs such as Risk Watch are knowledge, behaviour and environment.

Areas of Change

The first change that is measured is knowledge. This is the understanding the child has about the injury issues, including facts, figures, general information and procedures. For example, if a child learns the meaning of a traffic sign as a result of a Risk Watch lesson, there is a change of knowledge.

The second change that is measured is behaviour. This is the ability to perform a safety skill such as “get low and go under smoke to safety” or wearing a bike helmet correctly. The operative word is “correctly”. It means that the child can perform the skill as expected and when it is appropriate.

A change in knowledge doesn’t necessarily mean that there has been an accompanying change in behaviour. This is especially true in young children. Young children can recite a simple procedure such as “stop, drop and roll” yet not be able to perform the behaviour if their clothes catch on fire.

The third change that is measured is environment. A change in environment generally involves making the home, car, playground and other places safer (i.e. the installation of a smoke alarm or the use of shredded mulch or pea gravel under playground equipment).

Risk Watch is designed to measure all three changes. Written tests are used to measure changes in knowledge, while teachers can measure changes in knowledge, behaviour and environment after delivering Risk Watch lessons and using the evaluation indicator in the “Demonstrate Learning” and “Assessment” sections of the Lesson Plan. As well, the anecdotes shared by children and their parents or caregivers can offer insights into these changes.

Let’s discuss other types of change that can be measured. Impact evaluation is research conducted to determine how well a program is meeting its intermediate goals of changes in people’s knowledge, attitudes and beliefs. Outcome evaluation is designed to determine how well a program succeeded in achieving its ultimate objective of reducing morbidity and mortality². Outcome evaluation requires careful planning and execution and is probably not something you should plan on measuring at the end of your first year. If, after a few years, your coalition wants to measure outcomes, a word of caution: it involves thoughtfully designing and evaluation strategy and, most often, finding outside, professional help.

A final thought about evaluation. Evaluation is more than simply gathering scores, percentages and stories. Evaluation is the process of finding meaning from the numbers and information. A change in average pre- and post-test scores of five percent may not seem like much at first glance. But what if the scores were from a developmentally disabled class? What if the scores represented the highest gains the teacher had ever seen in any subject area? Make evaluation work for your coalition. Use the information you collect to tell a story, identify your “superstars” and promote your program. Don’t let it gather dust in some memo or report.

Here are some tips for effectively evaluating Risk Watch.

Tip #1: During your teacher workshop, spend plenty of time on the topic of evaluation. Your teachers need to know what is expected of them and how to use the right evaluation tools at the right time.
Tip#2: Use your evaluation data to improve your program. Share the data with coalition members, teachers and community leaders. Consider holding a news conference to announce the results.
Tip #3: Explain the evaluation results to students and caregivers and thank them for their support.
Tip #4: Solicit Risk Watch success stories from teachers (reports of students who have put their Risk Watch lessons into action). Each Risk Watch module includes an Incident Report Form. Encourage teachers to complete this form whenever one of the students has applied his or her Risk Watch knowledge in a real-life situation. A copy of the form should be sent to the Office of the Fire Marshal.

Summary

Does Risk Watch need to be evaluated in your community? Anytime you are initiating a new program and you want to make sure that learning is occurring, and anytime you are using other people’s money and resources, it is wise to do an evaluation.

Leadership Tasks

Identify the areas to be evaluated during your Risk Watch implementation effort.

If necessary, recruit outside assistance.

Talk to your teachers about the evaluation process and your expectations.

Make a plan to communicate your evaluation results.

²Thompson, Nancy J., PhD, and McClintock, Helen O.: Demonstrating Your Program’s Worth. National Centre for Injury Prevention and Control, Atlanta, GA. 1998.