Anytime you spend a lot of time in a project, you want to know that it is worthwhile. It’s always great when people tell you how much they enjoyed a class, but it’s even better to have statistics to tell you that the classes are making an impact. Of course, surveys and statistics, particularly in an uncontrolled informal education environment, can have validity issues. People responding to surveys often tell you want they think you want to hear, because they like you, or just want to be polite. Nonetheless, utilizing some type of evaluation to measure impact is important.
In our initial request for funding to develop the mobile cooking school, we knew we wanted to affect behavior by equipping people with skills and attitudes related to food preparation confidence and healthy choices.
As a starting point to request funding, before we had an evaluation instrument or even a curriculum design, we outlined the following objectives:
- After the completion of the 3 lesson series, 80% of participants will demonstrate increased ability to make healthy food choices
- After the completion of the series, 80% of participants will report increased confidence in cooking skills necessary to prepare healthy meals
- After completion of the series, 50% of participants will report an increased intake of weekly vegetables and fruits
- After attending at least one lesson, 80% of participants will report a better understanding of the MyPlate concepts
- After attending at least one lesson, 50% of participants will demonstrate a new food preparation skill that promotes good health
After the funding was granted and the curriculum design was envisioned, work on an evaluation instrument began. We were unable to find exiting instruments that would measure what we had in mind, so customized Pre and Post surveys were developed using examples from the Journal of Food Protection, Vol.67, No. 11, 2004, and the Journal of Nutrition Education and behavior, Vo. 43, No. 6, 2011.
In retrospect, as great as the objectives sounded in theory, we realize the wording of the outcomes doesn’t fit well within the constructs of the survey design. Having 80% of participants at a certain status is desirable; having an 80% increase is unrealistic (unless the audience knows almost nothing in the beginning). Because we are dealing with food and eating, which is a part of everyone’s everyday lives, attitudes and behaviors are going to be relatively positive in the beginning. With this in mind, even small increases toward the desired statuses really are victories.
By comparing responses of the pre and post survey and looking at the percentage of increase, the results are good, but not as large as the objectives initially stated. However, if we look at the post survey responses only, the stats are more in line with where we had anticipated seeing participants after taking classes.
In analyzing results from surveys taken in 2014-2015 and 2015-2016, comparing the increase of post responses over pre-survey responses, we see the rates of increase may seem small. I’m reminded of a book by Barrie Richardson called “The +10% Principle: How to get extraordinary results from ordinary people” The premise of the book is that although 10% may seem to be a small increase, who wouldn’t want a 10% raise in their salary, or a 10% increase grow of financial investment, or a 10% increase in employee productivity, and so on. Basically, a 10% increase is more than significant, it’s really extraordinary. When we think about the fact that many people who are willing to come to more than one class and complete a survey are people who are already interested in improving their health related skills, they are likely to come with fairly high rates of positive behaviors, so even small increases, around 10%, are pretty “extra-ordinary”. Classes like the mobile cooking school help those who are looking for solutions to the problems they have already identified for themselves.
Below is a summary of increases reported in post surveys as compared to pre-survey responses:
Behavior/Attitude | 2014/15 results | 2015/16 results | Cumulative
average |
Eating fruits and vegetable everyday (increase) | 8% | <1% | 4% |
Feeling confident in preparing fruits and vegetables (increase) | 6% | 7% | 6.5% |
Perception/dislike for cooking because too much time involved (decrease) | 9% | 9% | 9% |
Confidence in preparing meals from basic ingredients (increase) | 8% | 13% | 10.5% |
Perception of cooking as “too much work” (decrease) | 13% | 4% | 8.5% |
Understanding how food relates to health (increase) | 4% | 1% | 2.5% |
Confidence in preparing meals that follow USDA’s MyPlate (increase) | 18% | 29% | 23.5% |
Understanding of proper temperatures for food safety (increase) | 12% | 23% | 17.5% |
Preference for eating at home rather than fast food or restaurant (increase) | 13% | 5% | 9% |
Perception of ease of preparing food at home (increase) | 9% | 7% | 8% |
If we look at the results in a different way and consider the perceptions of participants after attending classes (i.e.: using only post responses only) we find:
Behavior/Attitude | 2015/2016 results |
I eat fruits and vegetable everyday | 89% (agree) |
I feel confident in preparing fruits and vegetables | 85% (agree) |
I dislike cooking because too much time involved | 57% (disagree) |
I feel confident in preparing meals from basic ingredients | 89% (agree) |
I think cooking is “too much work” | 57% (disagree) |
I understand how food relates to my health | 99% (agree) |
I feel confident in preparing meals that follow USDA’s MyPlate | 78% (agree) |
I understand proper temperatures for food safety | 99% (agree) |
I prefer eating at home rather than fast food or restaurant | 70% (agree) |
I think preparing food at home is easy to do | 81% (agree) |
In the event we make a new request for funding, I would maintain the same concept, but would certainly word the objectives differently to give a much better picture of the expected results. If we only wanted to look at behavior and attitudes after attending a class, a retrospective post survey would be all that is needed. One advantage of using a pre-survey is that is also serves as a needs assessment by asking participants what they would like to learn. This was useful in understanding the audience and tailoring classes to meet the groups needs and wants more specifically.
Here are some ways I would re-word:
After the completion of the one or more lessons/classes
- 80% of participants will report ability to make healthy food choices
- 80% of participants will report confidence in cooking skills necessary to prepare healthy meals
- 80% of participants will report understanding of the MyPlate concepts
- 50% of participants will report an adequate intake of weekly vegetables and fruits
- 50% of participants will demonstrate a new food preparation skill that promotes good health
Or if there is a preference toward focusing on increases, more realistic percentages would be more like the following.
After the completion of one or more lessons/classes
- 10% of participants will demonstrate increased ability to make healthy food choices
- 10% of participants will report increased confidence in cooking skills necessary to prepare healthy meals
- 10% of participants will report a better understanding of the MyPlate concepts
- 5% of participants will demonstrate a new food preparation skill that promotes good health
- 5% of participants will report an increased intake of weekly vegetables and fruits
Or, of course, totally new objectives can be designed to match funding requirements and/or another instrument. Either way, some type of evaluation is needed to make sure the project is on track and effective.
With this project in particular, a lot of time and effort is required and it reaches relatively small numbers at any one time (usually anywhere from 6 to 24 in a classes). Along with all of the other demands for Family and Consumer Sciences programming, I spend about 20% of my time on this project. I have co-workers and volunteers who help implement the classes. We have averaged about 30 classes, of which about 7 were series of 3, in 2015 and 2016 reaching around 150 people each year. Each class is about 2 hours long, with 2-5 hours of prep-time (planning, shopping, load, set-up, clean up, unload, etc.), and related travel time. This means lots of time has been invested but no huge numbers of contacts are made. Nevertheless, I am proud of the results. Seeing even small increases in the desired behaviors and attitudes and having large percentage of people practicing the desired behaviors after attending classes, gives me hope and makes me feel like the effort is worthwhile.
If you are interested in seeing a “Making a Difference” Texas A&M AgriLife Extension In-depth Summary report on this project, you can view mine here.