Visionaries in Evaluation: Beyond Samples & Quantitative Data
In my first post, I explained why we mustn’t let our fears prevent us from experimenting with evaluations that are neither purely quantitative nor based on random samples when trying to understand whether our social interventions are working.
Now, I want to share examples from visionaries who practice what I preach. The people you should know are Jim Henson, VI Hart, David McCandless, and Jonathon Harris.
Evaluations are a search for the truth about whether our work has changed lives. And I believe good evaluations are conceptually simple.
When Jim Henson was developing the design for Sesame Street, child psychologists and early education experts insisted that they knew the right way to do the show. Jim only cared about measuring whether each segment of video was good, and that kids were watching. You can’t teach kids anything if they aren’t paying attention, he argued.
His greatest innovation was a simple method to quantify whether kids were paying attention at each point during a test episode. They brought some five-year-olds into a play room and showed them visuals on two televisions simultaneously. One played an episode of Sesame Street, while the other played a random series of interesting images for seven seconds each ("the distractor").
Do the kids watch big bird talk about the letter "F" or is the image flip show of crickets, baseballs, old women, and street scenes on the alternate television more captivating?
By watching where each child’s gaze was focused every few seconds, they quantified something very abstract such as “interestingness.”
Suddenly you have a baseline-referenced measure of episode quality that also reveals which parts of episodes needed to be redone. Haven’t you ever wondered why there are image montages in between segments on Sesame Street? It's because kids respond to them.
And just a note on all those experts: They believed that mixing fantasy muppets with real people would lead to a child psychological meltdown. And they insisted that language needed to be taught the way it is in the classroom. Having ignored them, Sesame Street became the new model. It is produced in 30 languages and educates more preschoolers than all formal preschools in the world.
Education and art deserve to be sisters, as "mathmusician" Vi Hart has done with YouTube. She has published dozens of videos that illustrate the mathematical nature of order in the natural world. How does the Fibinaci sequence relate to flower petal arrangements? How can doodles reveal the fractal nature of number theory? And what does Pythagoreas have to do with the nature of sound waves?
This video is a better and simpler explanation of hearing than I gave my students in my Neuroscience course:
And how can we know that these artistic illustrations serve to educate students better than talking teachers? We could set up a classroom with me in one corner and Vi Hart’s video playing in the other, and I could prove how obsolete I’ve become as a lecturer…
Hart understands the essence of the concept on a musical, mathematical, visual, and neuroscientific level, and has integrated all of these into her ten minute video. What she omits is not essential to understanding. And the synthesis of all these perspectives serves to simplify the idea.
If only our evaluators aimed for such simplicity, and dared to ask whether our evaluations actually captivate decision makers any better than a TV screen flipping a bunch of random images.
Do these tools and stories really matter? It depends on the organization. Where we can get people and organizations talking, the projects that get implemented will be a closer reflection of the needs of the people—which means we can simplify our evaluations to just the one question: Did the project change lives?
And like I’ve explained elsewhere, truly great programs generate massive amounts of unsolicited positive feedback that leave traces on the web, in newspapers, in essays from college-bound students, and everywhere in between.
In Jim Henson’s experiment, impact evaluations did not require random sampling at all. A natural experiment emerged when some inner city TV stations ran Sesame Street and others ignored it. Years later, students who’d been raised on the show did better in school.
David McCandless and Jonathon Harris are two of the best “data artists” on the planet. They inspire me to build “insanely great” (to quote Steve Jobs) tools for evaluation that are simple, elegant, intuitive, and addictive.
Here are some examples of their work:
• Snake Oil Supplements (glean all the useful knowledge about vitamins/supplements from 10,000 peer-reviewed science articles in five minutes)
• I Want You to Want Me (building a 'mosaic of humanity' and understanding the nature of attraction between humans in five minutes)
Have you seen other creative ways of evaluating programs, projects, etc.? Share them in the comments below.