Think about the most fun you’ve had doing charity work. What was it that really appealed to you? Was it the smiling faces of kids playing a sport or painting a mural? Maybe it was the moment you realized someone’s life would be forever changed by the small token of love that a program enabled one person to give another.
Do you know what those moments have in common?
First, they are significant on an emotional, social, or metaphyiscal level—and so no traditional evaluation is well-suited to quantify them.
Second, these moments belong to those whose lives have changed. Your impact, as the person who helped make it happen, should not be the focus (unless you enjoy being self-centered and alone in the world).
So why do we continue to act as if “quantitative” surveys about our own “impact” are smart?
My decade working as a neuroscientist with actual “quantitative” data enables me to confidently dispel this notion once and for all. Here me out:
- Social change is social. That means it depends on people. Lots of them. People lie, especially in surveys, and often with the best intentions. Self-reports from people are not quantitative.
- So that’s why we have statistics, right? Inferential statistics depend on random sampling, and sampling is almost never random given the reasonable time and cost constraints placed on nonprofits.
- Even more alarming, statistics has no really solid way of telling if the sampling was done randomly.
- If random sampling is a problem, then results will not be reproducible over time and in different places. That’s why a lot of high-paid people interpret them and argue over methodology. But I think that’s a distraction from the core problem—which is our obsession with extrapolating from brief and tiny samples of life to broad and timeless descriptions of social change and impact.
- If you want quantitative data about people and social change, it’s probably more practical to transform our evaluation tools into a regular part of daily life—like Facebook or Google—so that we’re constantly looking at tens of thousands of bits of knowledge instead of just a few hundred.