Is Your Arts Programming Usable?

Posted by Mr. Ian David Moss, Oct 05, 2011 6 comments

Ian David Moss

At Fractured Atlas, we’re in the process of rolling out a few new technology products that have been in the pipeline for the past year or so. One of these is, which is the hosted version of the ATHENA open-source ticketing and CRM platform that was released earlier this year.

Another is a calendar and rental engine add-on to our performing arts space databases in New York City and the San Francisco Bay Area that will allow visitors to the site to reserve and pay for space directly online.

For both of these resources, we felt it was important to get feedback from actual users before proceeding with a full launch. So we engaged in a round of what’s called usability testing.

Usability testing differs from focus groups in that it involves the observation of participants as they actually use the product. So, rather than have people sit around a room and talk about (for example) how they might react to a new feature or what challenges they face in their daily work, you have people sitting in front of a computer and trying to navigate a website’s capabilities while staff members look over their shoulders and take notes.

Users are given concrete, specific assignments like “exchange Bob’s Friday night tickets for Wednesday night’s show,” and the degree to which they were successful, the time it took, and how they felt about it are all recorded as part of the testing process.

I was thinking recently about how the idea of usability testing could be applied to a marketing or programming context. After all, the purpose certainly translates: don’t you want to know how people really interact with what your organization offers? Especially if they’re not already extremely familiar with what you do?

They key feature of usability testing that makes it different from most other kinds of feedback-gathering methods is that it is based on direct observation rather than self-reporting. As applied to artistic programming, for example, it would be more like Drew McManus’ Take a Friend to the Orchestra series than Clayton Lord’s intrinsic impact initiative.

Because of its labor-intensive design and need for complicity from the study subject, it doesn’t generally allow for a large sample size, but the richly detailed information that comes out of it can yield tremendous insights.

Here are a few other scenarios that come to mind where usability testing (or something similar) could be relevant:

  • Can your administrative assistant’s grandmother find tickets to your latest show, without having been provided with your website or any other information about it?
  • How long does it take a visitor to your museum to find the gift shop and purchase a mug for her friend’s birthday?
  • Is a newcomer to your organization able to follow the plot of your opera’s performance of Rigoletto? How much, and what, does he remember about it afterwards?
6 responses for Is Your Arts Programming Usable?


October 09, 2011 at 5:33 pm

Hi Marcus,
There's really a wide range in the degree to which usability testers try to make their data objective. The "guerrilla" testing that Drew mentions above prioritizes convenience and cost over scientific exactitude. By contrast, old-school usability testing is very tightly controlled and more closely resembles a true experiment, but often at great cost.

A famous paper by Jakob Nielsen suggested that most obvious problems with a product are detected by just a handful of users, giving support to the more casual approach (which is also endorsed by usability expert Steve Krug).

  • Please login to post comments.
Marcus says
October 09, 2011 at 11:33 am

What measures do researchers who use usability testing take to ensure their observations are objective. You mentioned some factors but I am not totally convinced that an opinion would skew the data. Thoughts?

  • Please login to post comments.
October 05, 2011 at 12:35 pm

Ian, I love the idea of re-contextualizing "usability testing" as a feedback mechanism for arts audiences. As with online usability testing, the devil will be in the design, but the concept is well worth pursuing.

  • Please login to post comments.
October 05, 2011 at 3:00 pm

Great topic Ian and one often overlooked area for usability testing (on and offline application) is straightforward guerilla testing; meaning, grab a bunch of current users and have them use or do whatever it is you're working on. It helps sidestep many of the devilish details David is justifiably bringing up plus it produces much better "real" results than lab testing. Then there's the double bonus that it is less expensive and it forces those of us in the business to get some face time with our patrons. It's rarely a problem to get people to participate, who doesn't love to spout off opinions.

Even more interesting is passive guerilla testing; which would be more useful for non-online based testing. So if you have a new store layout, simply sit back and observe user behavior and make adjustments as necessary. Sure, there are terrific algorithms used for crowd management etc. But show me an arts group who has money to burn on that and I'll show you a group I want to send a proposal.

In order to plug some of the holes in guerilla testing, you have to revisit the issue from time to time to make sure the desired results are consistently reproduced; if not, then you may have missed something in the original analysis or conditions have changed. Either way, time for more testing.

Lastly, a bear trap to avoid if you're testing an online component is don't use your own laptops, tablets, or smartphone. Instead, let subjects use their own device so you can get a better sense of how different variations of the same interface can impact your design. In fact, whenever I do client demos for anything tech based service, I have them use their machine. Whether they realize it or not, it builds trust becasue they are using their own, familiar system (plus it means I don't have to haul my laptop around as much).

  • Please login to post comments.
October 06, 2011 at 3:16 am

nice article. it help me a lot. Try to do it my self.
Thanks for sharing...

  • Please login to post comments.
October 06, 2011 at 9:12 am

Ian - this is something so often overlooked in arts organizations. Because we in the arts are so comfortable with how things work, we sometimes get that insider "you're just supposed to know" attitude towards things, forgetting that newcomers don't have the knowledge base we all have. What you are suggesting is essentially taking the "secret shopper" approach so often used in competitive analysis and applying it internally. And Drew's point about guerilla testing is a good one. Doesn't have to be expensive and perfect to be useful.

  • Please login to post comments.