Evaluator: Vinson, M., Feng, L., & Lin, S. American Institutes for Research.

Evaluation Methods:

This evaluation gathered data from a variety of sources, including New Jersey Department of Education’s (NJDOE) data collection system to analyze program characteristics, student demographics, and attendance and outcomes data; pre- and post-program youth surveys; staff surveys; and New Jersey Standards Measurement and Resource for Teaching Data Warehouse (NJ SMART) to collect state test scores and unexcused absence data for 21st CCLC participants and non-participants. Propensity score matching was used to accurately compare the academic performance of students in 21st CCLC programs versus non-participants. In addition, correlational multilevel modeling techniques were employed to assess the relationship between participation levels in youth-reported program experiences and student outcomes. 

Evaluation Type: Experimental;Quasi-experimental

Summary of Outcomes:

This evaluation of New Jersey’s 21st CCLC programs during the 2016-2017 academic year found positive outcomes related to math state test scores for grades 4 and 8 and positive outcomes for English language arts state test scores for grade 11; reductions in unexcused school-day absences; and improvements in youth outcomes, including academic identity and interpersonal skills. 

The evaluation found modest outcomes relating to state assessments. Math scores for 4th and 8th grade participants were statistically significant, with those attending 60 or more days and participating in at least 30 hours of math-focused programming experiencing the greatest improvements compared to matched non-participants. Additionally, math scores for high-attending 4th and 8th grade participants–or those attending at least 45 days–were higher than their counterparts attending less than 15 days during the program year. Regarding ELA state test scores, 11th-grade students attending at least 30 days of the program and participating in at least 30 hours of ELA instruction in the program had the only statistically significant higher result than their non-participating peers, although the effect size was small. Related to academics, 85% of participants agreed that the program helped them learn things that will help them in school and 81% agreed that the program helped them think about the kinds of classes they wanted to take in high school.

Results relating to unexcused school-day absences were significant. Across nearly all grades, and for participants overall, regular attendees (30+ days) and high-attendees (60+ days) had lower unexcused school-day absence rates compared to matched non-participants, with the exception of 12th grade. When comparing the frequency of attendance between participants overall, those attending 45 days or more had a statistically significantly lower average unexcused absence rate than those attending less than 15 days. 

The evaluation also found positive outcomes related to youth experiences. In the post-survey, the majority of youth reported having the opportunity for choice within their programs, with most reporting that they sometimes or often get to choose how they spend their time (82%) and suggest ideas for activities (81%); a strong majority reported that they were able to help plan (64%) and lead activities in the program (61%). The overwhelming majority of participants also reported having positive relationships with adults in the program, with large shares of youth mostly or entirely agreeing that there is an adult that they enjoy being around (86%), helps them when they have a problem (85%), and has helped them find an interest or talent (78%). More than 8 in 10 participants agreed that the program helped them find out what they like to do (86%), what they’re good at (86%), and learn things that are important for their future (85%)

A correlational analysis of pre-and post surveys assessed four youth outcomes: identity, interpersonal skills, mindsets, and self-management. The analysis found that improvements in the four outcomes were correlated with the strength of relationships and opportunities for choice, as reported by youth, where the higher the youth rated their relationships in their program and ability to choose, the more they improved in academic identity, mindsets, self-management, and interpersonal skills.