“Not everything that can be counted counts, and not everything that counts can be counted.”  Undoubtedly, we’re all familiar with this quote, which is popularly but erroneously attributed to Albert Einstein.  (In fact, this statement first appeared in 1963 in Informal Sociology: A Casual Introduction to Sociological Thinking,” written by sociologist William Bruce Cameron.) 

Whether or not Einstein said it originally, there’s irrefutable truth in the statement. And, this idea causes us to think about two key aspects of evaluation and assessment: what’s the best way to measure what we believe we can, and how to deal with the “unmeasurable.” As I see it, the challenges we encounter when we try to measure the impact of our work don’t excuse us in any way from trying to get the information. Maybe the real truth isn’t that some things can’t be measured but rather that they can’t be assessed by conventional means, or by evaluation practices drawn from other sectors. We can’t be stymied by the reality that traditional methods of assessment don’t tell the whole story—or sometimes, even the best parts of the story. We have an obligation to be accountable to ourselves, our constituents, partners, peers, and supporters that we cannot shirk. We, as the community of arts and social justice practitioners, have to figure out how to judge our effectiveness, then how to inform those outside our sphere about our experiences.

Before delving into the “how,” let’s consider the “why.” The two most obvious reasons for assessment are 1) to learn how well we are accomplishing our organizational purposes, and 2) to honor our responsibility for accountability to ourselves, our constituents, and our supporters. Further, evaluation provides opportunities for us to strengthen practice, to improve strategies and programs for greater benefit. A final reason (for which we shouldn’t apologize) is that evaluation results are powerful aids for retaining or recruiting supporters and allies. In reality, if we are to achieve the impact we envision, we will need new resources for arts and social justice beyond those available from traditional arts donors. We have to cultivate funders and allies in other fields, particularly social justice.

The first step in being able to understand and convey the news of our successes must be the development of a common vocabulary to describe and discuss what we do. In the field, we don’t have a shared language we can use to talk about not only evaluation and assessment, but also about such fundamental concepts as the definition of arts and social justice. Creating vocabulary (and reaching consensus about meanings) would enhance both our internal conversations about issues, processes, opportunities and challenges and our communication with those outside our field.

With this common vocabulary, our ability to design and implement evaluation processes (that include the appropriate involvement of internal and external constituents and the right mix of qualitative and quantitative data collection) that suit our work is significantly improved. We can share experiences in a more parallel way, and present results that reveal a degree of consistency about the state of the arts and social justice realm. But we have to be diligent to avoid the pitfalls of using evaluation methodologies that really don’t fit what we do. We must be particularly careful about the use of metrics.

In the cultural sector, attempts to apply the performance management concept of “metrics” occur too often, and usually without due consideration for the characteristics that distinguish us from profit-making enterprises. The over-reliance upon metrics is often encouraged by funders who expect that the statistics we provide will support their own assessment need, i.e., to demonstrate that their grantmaking is advancing their philanthropic goals.  In other cases, arts organizations themselves embrace metrics in an attempt to demonstrate the value of their programs and services. Counting is easy.

But metrics alone don’t convey the impact of the work of arts organizations; they’re incomplete and reductive. Metrics most often seem to emphasize what organizations find immediately measurable (attendance figures and ticket sales, constituent demographics, etc.) — even if those are low value in the context of long-term impact — and tend to ignore high value measurements (changes in community participation, the effects of artists’ work on societal movements, policy and legislation, etc.,) simply because they seem harder to measure. This phenomenon, known as “measurement inversion,” can lead to conclusions about organizational effectiveness and worth that are based on only partial information.

We in arts and culture need to figure out how to measure impact. We need to be proactive, rather than being responsive to funders or resorting to just offering statistics. We have to develop our own measurements, and teach others what counts. Of course our measurements should be both quantitative and qualitative, both metrics and stories.

The creation of common language combined with data collected using evaluation methods and tools appropriate for work in arts and social justice will help us build a strong case that we can use to cultivate new allies. A compelling case heightens our legitimacy and worthiness to those from whom we seek financial support and strategic partnerships. It is also key to making our work legible to those in other sectors.  Since our work is not generally known in other arenas, our success depends on our helping practitioners in other industries understand how arts and social justice works, and our presenting solid evidence of our success that we collect through evaluation.

In the end, conducting evaluation should not subvert our lived experiences and instincts about the power of arts and social justice. Rather, it should affirm and inform our shared work. Evaluation and assessment are critical to the vitality and vibrancy of the arts and social justice field. They are key components for our own understanding of the long-term impact of our work. Data we gather through these processes form the essential core of the case we need to make to new allies, especially those unfamiliar with what we do. Developing a common vocabulary is the first, essential step.