Our Approach to Evaluation Should Be Just as Provocative as Our Practice

Posted by Jason Yoon, May 03, 2012 3 comments

Jason Yoon

One of my first “real” jobs was as an art specialist at a start-up charter elementary school. We did a lot of grading. The school was developing a comprehensive academic scope and sequence. Report cards reflected maybe 100-some skills and standards by subject. Teachers spent hours assessing each student.

As an idealistic young educator, the complexity of the thing was actually exciting. I couldn’t wait to see my “enrichment” section of the report card and the skills and standards in the arts I was responsible for. I then found that I had the smallest section of the report card:







4=Excellent 3=Good 2=Needs work 1=Seriously deficient

That’s it?

This school had mapped skills and standards to the minutest details and I only got two vague behaviors? I wanted credit for teaching my kids important and real things too!

I bring this up not to criticize the school. The school has expanded admirably since, received national recognition, expanded their arts programs and I figure now has a more robust method for assessing arts learning.

In that small example, is the dilemma that faces the art world right? We want to be taken seriously.

And one message is that we can get there by being graded and measured in easy-to-digest numbers like other subjects or fields. The institutional message then was that I was just the art teacher. Put simply, the school’s charter probably wasn’t going to be revoked if my kids couldn’t paint.

But we have to be careful not to adopt the fallacies of the “accountability” movement, too.

Instead of taking the worst lessons of other sectors in evaluation, research, accountability, measurement (or whatever terms you want to use), we in the arts have a responsibility to LEAD, not follow in developing new ways of finding and sharing knowledge and insight.

Isn’t that what we do in the arts? We see new and exciting ways to do things. We break paradigms, we challenge prevailing wisdom, we provoke. We make trouble. Our approach to “evaluation” should be just as provocative as our practice.

And why look to the accountability movements of other sectors?

Short-term and narrow quantitative performance measures contributed to the collapse of our global economy, we have falsified test scores, and potentially unhealthy quotas for arrests and police stop and frisks.

We can debate the particulars of each of these controversies, but overall it’s a troubling pattern. It indicates the problem with applying supposedly rigorous business world methods to complex human work.

More troubling, these instances involve seemingly straightforward quantitative indicators of performance: test scores, stock price, and arrests. Yet we still have very real ongoing questions about what any of it means. And that’s not even talking about the less measured disparate impacts (communities of color suffering from aggressive policing, children receiving less art and science in test-focused schools, unchecked financial engineering etc.).

In Policy Paradox, Deborah Stone writes about the dangers and unintended consequences and incentives that arise from the use of performance measures in public policy. It’s also big business.

What is the middle ground between rigor and authenticity in the arts? How do we know what we’re accomplishing?

Maybe we just ask.

Years ago, at the end of a talk with the evaluation director of a major national foundation, I asked him about “evaluating our impact.” He asked, “How many kids do you have in the program?”

“About 12.”

“Do you know them well?”

“I think so.”

He told me to just call each of my students at the end of the cycle, to have an open-ended conversation about what they got out of the program and what could make it better. It was only 12 students, I should be able to get to them all and if I couldn’t that was a sign of something too. “Don’t over think it.” Just ask.

So, regarding this question of social impact, on a bigger scale, that’s what we’re working on at New Urban Arts.

More specifically, through the Ford Foundation ARTOGRAPHY program, we’re currently working with our founder, Tyler Denmead Ph. D., on an ethnographic arts research framework for our community studio.

We’re pulling from lots of different sources and fields for inspiration, Sarah Lawrence Lightfoot, Maribel Alvarez, one of the ARTOGRAPHY documentarians, the growing field of publicly engaged scholarship, ethnography, participatory research, and more. We’re also including with this an exhaustive literature review relevant to our context as a community youth art studio.

We’re not sure what our framework is going to look like yet, but our efforts are coming out of a deep dissatisfaction with the status quo that we feel is presented to us as practitioners. If we choose not to evaluate in a traditional sense, we are irresponsible managers unaccountable to our publics. But if we embrace this performance measurement and accountability framework, we start to kill what makes our spaces so relevant in the first place.

I argue that there are multiple paths to being rigorous, about knowing your work and being accountable. But we want to own “rigor” and “accountability” on our terms and in new and creative ways and most importantly honor first voice, through a framework for our diverse participants “to describe and interpret what they do and why it is meaningful to them.

Traditional impact evaluation is about proving something happened, that some change in condition occurred as a result of some intervention you’ve isolated, but the reality is that “proof” is elusive. Millions of dollars have been poured into trying to prove things perhaps somewhat dubiously.

What we’re interested in is sharing how our community comes together to generate new knowledge and to share that knowledge through the voices of our community.

To follow our progress, please join us at www.newurbanarts.org.

3 responses for Our Approach to Evaluation Should Be Just as Provocative as Our Practice


May 04, 2012 at 9:08 am

Jason - you connect the dots perfectly between two current realities - evaluation practices that fail and imperil our young people, and our culture's apathy (and ignorance?) about the value and necessity of the arts in our lives. Results are all around us, as you say, and we at Merge contend that first is a direct result of the mentality of the second ... national ignorance has devalued the importance of ongoing creative human development, which is so beautifully accomplished through the arts.

You also correctly point out that - "millions of dollars have been poured into trying to prove things perhaps somewhat dubiously" - and we see that the dismal results were precisely because the evaluation methodologies were developed by the same thinking that dismisses the arts. Evaluating and focusing on a very narrow set of "skills" has resulted in a society that is suffocating for lack of oxygen.

After spending almost two decades running a community arts mentoring nonprofit, we have developed an evaluation software that assesses meaningful qualities and skills which are prerequisites to true personal success - whatever that means for each individual. It was developed by my highly creative husband/partner working in tandem with research scientists, social workers, clinicians, and software developers. It took years of development because it evolved organically and we kept at it until it was truly essential.

Perhaps you could say it's representative of America's "new intelligence". I invite you to take a look at http://merge-education.com/evaluation-software.php and would be delighted to give you a virtual tour.

  • Please login to post comments.
May 16, 2012 at 9:14 am

A PS from Merge Education: Since posting this comment we've gotten lots of interest in SETS - we're in the middle of a site re-vamp to totally new, so please contact us or check back mid-June. Thanks!

  • Please login to post comments.
July 02, 2012 at 2:35 pm

Merge Education's new site is now live, representing the new evaluation software far better. Thanks again to all who expressed an interest!

  • Please login to post comments.