Our Approach to Evaluation Should Be Just as Provocative as Our Practice
One of my first “real” jobs was as an art specialist at a start-up charter elementary school. We did a lot of grading. The school was developing a comprehensive academic scope and sequence. Report cards reflected maybe 100-some skills and standards by subject. Teachers spent hours assessing each student.
As an idealistic young educator, the complexity of the thing was actually exciting. I couldn’t wait to see my “enrichment” section of the report card and the skills and standards in the arts I was responsible for. I then found that I had the smallest section of the report card:
4=Excellent 3=Good 2=Needs work 1=Seriously deficient
This school had mapped skills and standards to the minutest details and I only got two vague behaviors? I wanted credit for teaching my kids important and real things too!
I bring this up not to criticize the school. The school has expanded admirably since, received national recognition, expanded their arts programs and I figure now has a more robust method for assessing arts learning.
In that small example, is the dilemma that faces the art world right? We want to be taken seriously.
And one message is that we can get there by being graded and measured in easy-to-digest numbers like other subjects or fields. The institutional message then was that I was just the art teacher. Put simply, the school’s charter probably wasn’t going to be revoked if my kids couldn’t paint.
But we have to be careful not to adopt the fallacies of the “accountability” movement, too.
Instead of taking the worst lessons of other sectors in evaluation, research, accountability, measurement (or whatever terms you want to use), we in the arts have a responsibility to LEAD, not follow in developing new ways of finding and sharing knowledge and insight.
Isn’t that what we do in the arts? We see new and exciting ways to do things. We break paradigms, we challenge prevailing wisdom, we provoke. We make trouble. Our approach to “evaluation” should be just as provocative as our practice.
And why look to the accountability movements of other sectors?
Short-term and narrow quantitative performance measures contributed to the collapse of our global economy, we have falsified test scores, and potentially unhealthy quotas for arrests and police stop and frisks.
We can debate the particulars of each of these controversies, but overall it’s a troubling pattern. It indicates the problem with applying supposedly rigorous business world methods to complex human work.
More troubling, these instances involve seemingly straightforward quantitative indicators of performance: test scores, stock price, and arrests. Yet we still have very real ongoing questions about what any of it means. And that’s not even talking about the less measured disparate impacts (communities of color suffering from aggressive policing, children receiving less art and science in test-focused schools, unchecked financial engineering etc.).
What is the middle ground between rigor and authenticity in the arts? How do we know what we’re accomplishing?
Maybe we just ask.
Years ago, at the end of a talk with the evaluation director of a major national foundation, I asked him about “evaluating our impact.” He asked, “How many kids do you have in the program?”
“Do you know them well?”
“I think so.”
He told me to just call each of my students at the end of the cycle, to have an open-ended conversation about what they got out of the program and what could make it better. It was only 12 students, I should be able to get to them all and if I couldn’t that was a sign of something too. “Don’t over think it.” Just ask.
So, regarding this question of social impact, on a bigger scale, that’s what we’re working on at New Urban Arts.
We’re pulling from lots of different sources and fields for inspiration, Sarah Lawrence Lightfoot, Maribel Alvarez, one of the ARTOGRAPHY documentarians, the growing field of publicly engaged scholarship, ethnography, participatory research, and more. We’re also including with this an exhaustive literature review relevant to our context as a community youth art studio.
We’re not sure what our framework is going to look like yet, but our efforts are coming out of a deep dissatisfaction with the status quo that we feel is presented to us as practitioners. If we choose not to evaluate in a traditional sense, we are irresponsible managers unaccountable to our publics. But if we embrace this performance measurement and accountability framework, we start to kill what makes our spaces so relevant in the first place.
I argue that there are multiple paths to being rigorous, about knowing your work and being accountable. But we want to own “rigor” and “accountability” on our terms and in new and creative ways and most importantly honor first voice, through a framework for our diverse participants “to describe and interpret what they do and why it is meaningful to them.”
Traditional impact evaluation is about proving something happened, that some change in condition occurred as a result of some intervention you’ve isolated, but the reality is that “proof” is elusive. Millions of dollars have been poured into trying to prove things perhaps somewhat dubiously.
What we’re interested in is sharing how our community comes together to generate new knowledge and to share that knowledge through the voices of our community.
To follow our progress, please join us at www.newurbanarts.org.