Reevaluating My Thinking Around Evaluation
When I asked the bloggers for Animating Democracy’s Evaluation & Social Impact Blog Salon to write about this topic, I thought I knew what I was going to get. Animating Democracy’s Impact Initiative has been going strong for several years now with a fantastic set of evolving framings, vocabulary, metrics, methodologies, etc. and so forth.
In addition, a good handful of these folks have worked with us before and the rest I know through their incredible creative work. None of them are strangers to questions of social impact or evaluation.
I expected discussions about how to show funders and community leaders what impact was made; talk of how to establish outcomes and indicators; examples of surveys and interviewing. While many of these were touched on throughout the salon, what did emerge made me start to adjust my thinking around evaluation.
Perhaps even more important was what surprised me: that the trends in evaluative thinking and practice came from the work itself rather than an external driver.
Is storytelling just a fad?: Qualitative v. quantitative
At Americans for the Arts, we make the case for the arts daily and within that there is a delicate balance between the concreteness of numbers and the power of stories. However, as noted by Katherine Gressel, “surveys and statistics are out; stories and experiences are in.”
While I would argue that there’s still merit in both deep and broad collection of feedback, I think that storytelling is more than just a fad. Simply put: using storytelling, the evaluative method mirrors the work instead of imposing on it.
On the output side, storytelling has its own power. The acknowledgement of this trend is not simply constrained to social impact evaluation, nor even the field of arts for change work. In a realm like evaluation, where anything simple is golden, using storytelling for feedback (input) and casemaking (output) makes sense, particularly if it more accurately conveys the positive change being made.
Art in life: Evaluate in life
Although alluded to already, another theme throughout the salon was the idea that art is part of life; evaluation should follow suit.
Mark Stern spoke about the capabilities approach, the idea that social well-being is a product of people’s opportunities to be and do in certain ways, which could serve as a framework to evaluate the arts within a healthy cultural ecosystem that includes all the dimensions of social well-being.
This concept was exemplified by the Inocente, documentary mentioned by John Bare. I couldn’t summarize it better than when he wrote: “there is only so much to be gained by pitting the arts against other independent variables on tests of short-term metrics. There is something bigger out there: We can identify the situations where arts contribute to sweeping social change.”
For all the discussion in the public art field about the need for more comprehensive/rigorous evaluation models, I was inspired that some of the most concrete examples of making evaluation mirror art and life came from Penny Balkin Bach and Rachel Engh’s posts about public art projects incorporating technology that both added to the art experience and provided evaluative feedback.
A matter of timing?: Before, during, after...and after
A couple of weeks ago, I was privileged to sit in on the final presentations for a Virginia Tech course on evaluating public art. One of the aspects that I found most interesting was that different groups did their evaluation at different points in the public art’s timeline (i.e., before, during, and immediately after).
Likewise, one thread in this blog salon that stood out to me was the idea that evaluation doesn’t and shouldn’t have to be a one-time deal. Chris Dwyer’s post about beginning to see concrete indicators of The Shipyard Project over 15 years later speaks to the challenge of tracking outcomes that might appear 5, 10, 30 years down the line.
Equally important, evaluative work has a possible role to play at the beginning and middle of a project’s lifespan, which leads me to...
With work that stems from a desire to address bigger social needs, it’s not enough to simply tack on a survey at the end of a project. I’m inspired by these posts because so much of the thinking in them around evaluation comes from the work itself and the desire to make positive change rather than from external forces like funders or public officials.
In the end, it might be about creating a methodology, nagging people about surveys, mining data, and collecting stories, but the bigger, more interesting challenge is how that evaluation will shape what we do next.
Thanks to everyone who was following, sharing, and commenting on the blogs. You can view all of the Blog Salon posts using this link.
And, a huge thanks to our amazing, innovative, and articulate bloggers for sharing your perspectives and making me reevaluate my thinking around evaluation.