How To Test Content: ConFab 2011

Testing Content

Session description: Testing the usability of interfaces is second nature for many user experience professionals. But how can you make sure that the content you’ve written for that interface will also be usable? With techniques drawn from the usability and education fields, you can estimate the difficulty of text and test it with users to make sure it’s understood.

Speaker: Angela Colter, principal of design research at Electronic Ink.

Quote marks indicate direct quotes. If there aren’t quote marks, it’s paraphrased.

The skinny

It’s pretty clear that the folks at this conference are most interested in a) meeting their content strategy idols (is Erin Kissane Vishnu reincarnate?) and b) measuring ROI of content strategy. At least, I’m going to assume that’s why there’s all the “I met Erin!” namedropping going on and why all of the data sessions are packed, like this one.

What she said

Ever read credit card fine print? When credit card debt was overwhelming Americans, Angela was tasked by the United States government to find out if the fine print had something to do with it. Was it readable and understandable by the average consumer? She had to test it and report back to the Government Accountability Office. What was the first thing she thought? “Oh, shit.”

Over time, tests for content readability have emerged, and there are better ways to know whether people understand content. We’re going to talk about three: readability formulas, usability tests and Cloze tests.

Readability formulas

Readability formulas are tests that measure things such as words, sentences or paragraphs and often report results as the reading grade level of text. Here’s a common formula:

(.39 x #words/#sentences) + (11.8 x #syllables/#words) – 15.59 = grade level.

Most readability tests have a semantic (word level) aspect and a syntactic (sentence level) aspect. But when you’re talking about readability formulas,you’re only counting the things that can be counted. This means that you can take a sentence that makes sense, jumble the words around, plug it into a readability formula and still get the same grade level. So why still use them?

• They’re good at quickly estimating text difficulty, which can show red flags.
• They produce an easily understood metric for clients.
• They can help you argue for funding to test with users.

That said, know that:

• They are not magic.
• They should not be your only evaluation method.
• They are not a guide for writing good content.

Usability tests

The whole point of usability testing is to find out what users do, not what they say they do. With testing content, you want to test what they understand, and not what they say they understand.

How do you design a usability test? There a lots of good resources, but this is Angela’s advice:

• Recruit the right participants.
• Identify critical issues to user and business.
• Create tasks that test those issues.
• Ask user to think aloud, to paraphrase. If they can’t put it in their own words, they don’t really understand.
• Know what the correct answer looks like. If you’re a consultant and not an expert on the content, you need to know the right answer yourself.

While you can create a usability tests with very specific tasks—e.g., go to a transportation website and “Buy a Zone 2 Trailpass”—you’re often asking people to skip over content that might be relevant to them. So a better way to phrase the task of “Buy a Zone 2 Trailpass” is “Buy a pass that suits your needs.” This way, you’re not just testing the interface, you’re testing the content.

Why use usability tests?

• They tell you what problems user have.
• They reveal why the user is having problems.
• They make you feel the user’s pain.
• They can end in-house opinion-based arguments.

That said, they can require lots of resources and money.

Cloze tests

Cloze tests are based on the Gestalt theory that your brain automatically fills in missing information. By removing every fifth word of a paragraph, you can measure understanding.

Here’s how to create a Cloze test:

• Take at last a 125-word sample of text.
• Remove every fifth word.
• Go for 25-50 blanks.
• Ask users to fill in the exact word the author used.
• Calculate the percentage of exact matches.

Then, score it. Basic Cloze test scoring percentages break down like this:

• 60% or more: appropriate (reader is functioning at an independent level)
• 40-60%: may need additional help (reader is working at an instructional level)
• 40% or less: too difficult (content needs to be rewritten)

[Audience participates in a Cloze testing example from, dang it all, the credit card fine print. Surprise, surprise... everyone scored at 40% or less. The copy was too difficult and needed to be rewritten.]

Why are Cloze tests so valuable?

• They indicate whether material is appropriate for a given audience.
• They take into account reader’s prior knowledge of the subject matter.
• They take little time to create, administer and score the test.
• They do not require subject matter expertise.

How do these tests compare?

Readability formulas: You need just one person (you), they’re very quick, they’re low cost and they test text.

Cloze tests: You need as many people as it takes to convince you (5+), they’re quick, they’re in the middle of cost ($0-$25 per participant) and they test text.

Usability testing: You need as many people as it takes to convince you (5+), it’s not quick, they’re at the high end of cost ($100+ per participant) and they test text and wireframes.

Audience participation

Have you done any testing of videos? No, so I don’t have anything to add to that conversation.

I’m writing something that will be testing in 10 different languages. There will be testing at the end, but are there different readability formulas I can use now? I have only worked with readability formulas for English language. I know that there are some out there. (Another audience member comments that the one he used for French didn’t work.) I would say that it would be better to do a usability or Cloze test in English before it gets translated. Give it your best shot with the original.

How do you deal with synonyms in Cloze tests? Do you do have you use every fifth word? There are something like a thousand studies on how you implement a Cloze test. You can choose every n-th word, or only the important words, you can accept exact matches, or accept synonyms. “Just pick one and stick with it and you’ll get good feedback.” Be consistent. I like exact matches, because I don’t have to make those decisions about what synonyms match—it’s faster. And you don’t introduce that level of subjectivity into it.

For more resources on these specific tests, see “Testing Content” on A List Apart.