Mark Rodriguez

Upon reviewing a blog entry about The Arts and Achievement in At-Risk Youth study released by the National Endowment for the Arts (NEA) earlier this month, I ran across a respondent who stated, “It’s great to have all of these studies, but how does it help me and my organization? How can small or midsized arts organizations measure their impact without the resources of large institutions like the NEA?”

The following shares the story of how Changing Worlds, a midsized Chicago-based educational arts nonprofit went from basic surveys and pre- and post-residency exercises to a longitudinal study that improved our practice, reaffirmed the quality of our program, and helped build an organizational culture of inquiry.

In 2003, I became the executive director of a small start-up nonprofit that had little to no infrastructure in place to assess its programs. We had lots of informal data and some feedback from program partners. I knew immediately that if we were going to grow, thrive and succeed, we had to identify our unique niche, solidify our program model and select program inquiry questions we wanted to explore.

From 2003–2008, we went through various renditions of evaluation tools and we even contracted with three independent evaluation consultants. After five years, we learned some new things, developed the basic capacity to measure the impact of our residency programs and invested lots of time. While this helped us gain insight into our short-term impact, it didn’t address the potential long-term impact and implications of our program.

In 2009, we set out to do just that, measure the long-term implications and impact of our in-school integrated arts, cultural awareness and literacy residency on students. Given the program’s unique structure, with its combination of writing, arts, and cultural awareness activities, we were looking for answers that could not be found by looking at similar programs.

We started by scaling back to scale up. We decreased the number of in-school residencies we had in order to invest those resources in evaluation efforts that could help us measure our long-term impact.

We conducted a request for proposals, seeking evaluation and school partners that would embark on a three-year journey with us. In 2009, we partnered with three Chicago Public Schools and the Center for Urban Research and Learning (CURL) at Loyola University to launch our study.

We began in the 2008–2009 school year, working with groups of fourth graders and tracking them over the course of three years. We used data collected from sixth graders during the program launch year at each school to serve as a delayed comparison. We used all of the same assessments tools with the comparison group and planned to compare the results with our students when they reached the sixth grade.

Students taking part in a Changing Worlds activity.

The results were greater than we had anticipated. Students in our program outperformed comparison group students in all areas measured.

While we recognize that our program isn’t the sole factor connected to academic outcomes, we found that test scores for students in Changing Worlds’ program were greater than control group students, with an average positive difference of 11.5 points percentage points on standardized test across all schools. A review of test score data over time revealed that Changing Worlds’ students who were performing at a lower level than nonparticipants caught up after one year of being involved in the program. Over the three year period, Changing Worlds’ students continued to outperform nonparticipants at higher percentage levels each year.

While the quantitative data is critical, the many things we learned from the stories we collected from students and their families were equally important and rewarding.

“The program opened my eyes to new ways to express myself.”
”The program helped me learn more about my peers and how to work together.”
“The program taught me how to look at things from different point of views.”
“The program changed my life.”

These quotes from students highlight the not-so-measureable social impact and lifelong skills students gain from the arts.

While the academic point gains, increase in writing scores, and ability to demonstrate greater artistic capacity is important, the jewels are in the student feedback and the value of the arts that goes beyond a test, but leads to the global, innovative citizens of tomorrow.

For more information on “Unlocking Pathways to Learning,” our three-year study, visit www.changingworlds.org.