Afterschool Impacts Database
Evaluation 101
Learn more about the basics of evaluation, as well as how to find an evaluator for your program.
Afterschool Evaluation
View Afterschool Alliance resources, including a glossary of terms used in the database, evaluation-related blogs, webinars and more.
Additional Resources
Our list of evaluation resources from other organizations, including how to collect and work with data.
Afterschool Research
Want to find what we know about afterschool programs more broadly, not just individual programs? Head to our Afterschool Research page!
4-H Water Wizards: Lessons learned for effective afterschool science programming
Year Published: 2022
This study found that students who participated in the Sacramento 4-H Water Wizards program from 2012-2016 had a better understanding of water science and a heightened awareness of water issues and conservation behavior. Program staff reported a higher level of understanding around content knowledge (different aspects of water science) and teaching pedagogy (teaching science in an inquiry-based way), as well as increased enjoyment of science. Evaluators found that including training sessions, teaching materials, and coaching opportunities throughout the program for staff helped contribute to the program's success, and 80 percent of staff said they would teach the program again.
Program Name: Sacramento 4-H Science Afterschool
Program Description:
The 4-H Water Wizards project is a 12-week program for students in 4th through 6th grade, developed by the University of California's 4-H Youth Development Program to provide science programming outside of the school day and develop young people’s understanding about and decision-making around water use. The curriculum takes a hands-on inquiry-based approach and includes 11 sessions around water and the environment, water properties, and a service learning project on a water-related issue.
Scope of the Evaluation: Local
Program Type: Afterschool
Location: Sacramento
Community Type: Urban
Grade level: Elementary School
Program Demographics:
From 2012 to 2016, 49 percent of 4-H Water Wizards Project participants were male, and 51 percent were female. Thirty-eight percent of participants were Hispanic, 23 percent were African American, 16 percent were White, 15 percent were Asian, 6 percent were Pacific Islander, 1 percent were Native American, and 1 percent identified as Other. Program staff from 2012 to 2016 were 78 percent female and 22 percent male, and 35 percent were Hispanic, 16 percent were Native American, 16 percent were African American, 16 percent were White, 6 percent were Asian, 4 percent were Pacific Islander, and 1 percent identified as Other. At least half of students in schools served qualified for free or federal reduced-price lunch.
Program Website: https://sac4h.ucanr.edu/Programs/4-H_Science_After_School/4-H_Water_Wizards/
Evaluator: Bird, M. & Subramaniam, A. University of California Cooperative Extension
Evaluation Methods:
To measure student outcomes, evaluators created pre and post-program surveys to measure differences in students’ attitudes and knowledge around water use, water conservation, and how much they enjoyed learning about science. For staff, evaluators developed two surveys administered at the first and the final training sessions to determine knowledge and behaviors around teaching science. In all, data were collected from 469 students and 59 staff across 21 afterschool program sites using the 4-H Water Wizards curriculum. Evaluators conducted paired sample t-tests for each group of surveys to determine if there were any statistically significant differences. They used a grounded-theory framework to analyze and compile common themes from the short answer question.
Evaluation Type: Quasi-experimental
Summary of Outcomes:
Researchers at the University of California Cooperative Extension evaluated the 4-H Water Wizards program to explore its impact on students and staff. Students participating in the Sacramento 4-H Water Wizards program between 2012 and 2016 demonstrated a better grasp of water science and a greater awareness of water-related issues and conservation practices. The program's staff also reported an improved understanding of content knowledge (such as the water cycle), teaching pedagogy (how to teach science using inquiry-based methods), and increased enjoyment of science. The study's evaluators emphasized the importance of including training sessions, teaching materials, and coaching opportunities for program staff.
Students who participated in 4-H Water Wizards had a positive experience, with more than 4 in 5 respondents (85 percent) sharing that they enjoyed the program. Furthermore, general awareness of water issues and knowledge about water science grew for most students participating in the program. While there were some differences by age, combining results across the four years of the study, mean post-test scores were significantly higher than pre-test scores, and looking at each year individually, post-test scores were also significantly higher. Regarding differences by age, students in fourth and fifth grade showed a statistically significant increase in knowledge around water science, while those in sixth grade did not have a statistically significant increase in knowledge. Evaluators posit that the sixth-grade results may have been partially due to a smaller sample size and higher pre-program survey scores. Moreover, most participants (79 percent) shared that they used less water after participating in the program and had a higher likelihood of identifying two or more water-related issues in their communities in the post-test questionnaire compared to the pre-test.
Evaluators found a significant positive change in staff outcomes regarding their ability to teach the content and their confidence in teaching the content. Staff received three training sessions, materials, and coaching throughout the program. Training allowed staff to experience the content they would be teaching and to practice inquiry-based teaching. In nearly every category asked about, program staffs’ assessment of their skills and understanding of a content area was higher in the post-program survey compared to the pre-program survey. For example, on a scale from 1 to 4, where 1 equals “poor” and 4 equals “very good,” staff reported growth in their skills and attitudes about teaching science (2.3 vs. 3.0), understanding science (2.6 vs. 3.2), and enjoyment of science (3.1 vs. 3.6). Additionally, staff felt more confident in teaching aspects of the curriculum, such as the water cycle, properties, issues, and conservation, and, more broadly, inquiry-based teaching. Staff enjoyed teaching the curriculum, with 4 in 5 (80 percent) responding that they would teach 4-H Water Wizards again.
Regarding challenges, nearly 3 in 10 staff (29 percent) shared that juggling time constraints, like homework and holidays, was the most challenging part of delivering programming. The timing of additional staff training sessions was a barrier, with some staff reporting that changing the training time from evenings following programming would be beneficial.
Date Added: February 22, 2024
