One of the key concepts of WCAG 2.0 is testability. This is about ensuring that each success criterion can be tested either by a machine, or manually in such a way that at least 80% of the human testers would agree whether it is a success or a failure.

Testability Costing Too Much?

Gian Sampson-Wild has written an article for A List Apart about this, entitled Testability Costs Too Much where, after explaining what testability means in terms of WCAG 2.0, is then critical of it, arguing that a number of the current guidelines aren’t actually that testable — it’s quite difficult to test whether the meaning of alt text accurately conveys the same information as the image it represents.

The key point that she’s making can probably best be summed up by her comment in a WCAG teleconference:

if you lock out guidelines [because] we can not define them in a testable manner, then we run the risk of locking out guidelines that people find useful and that increase the accessibility of contentGian Sampson-Wild

One of the admitted weaknesses of WCAG 2.0 as it currently stands is in the area of cognitive disability, and it is also this area that would probably be best served if WCAG 2.0 success criteria did not have to be testable, as concepts such as “use the clearest and simples language appropriate for a site’s concept” or “divide large blocks of information into manageable groups where natural and appropriate” are easy to understand and implement, but difficult to test in a consistent manner.

Rather unfortunately to my mind, rather than continuing this tack, Gian then criticises the WCAG Working Group, saying that her status as a member in good standing of the working group was revoked due to non-participation and implying that this was precisely because of her stance on testability.

Her argument regarding testability either holds water on its own or it doesn’t, irrespective of the actions of the Working Group. The implication that her ‘good standing’ status was revoked because of her stance on testability is a somewhat odious one, but the fact that she herself admits that she had not met the participation criteria rather sinks her argument.

I understand her personal difficulties in participating enough to qualify — indeed while I would like to take part in something like the WCAG Working Group I doubt I would ever have the time to qualify as “a member in good standing” – but the very fact she did not meet the publicly stated criteria means that the Working Group were specifically supposed to revoke her ‘good standing’ status.

She charges that not everyone is treated in this manner: a charge that if true ought to be addressed as it would indeed suggest at unfairness. Indeed as I have heard a number of criticisms from a number of people (although probably about half of them from Joe Clark) about the way in which the WCAG Working Group operates, it wouldn’t surprise me if there are problems with it that ought to be addressed. The point nevertheless remains, had she met the criteria they could not have revoked her status (or at least not without needing some other reason!). She didn’t keep her side of the deal.

Testability, the WCAG Samurai Errata and cognitive disability

Gian also holds up the WCAG Samurai Errata (an unofficial ‘patch’ for WCAG 1.0 created by Joe Clark and other persons unknown) as an example of guidelines which are clear, don’t rely on the concept of ‘testability’ and are useful to developers.

Well, yes. The WCAG Samurai Errata are indeed all of those things. However, they aren’t the answer to the issues of cognitive disability, and nor are they perfect. I find I don’t agree with all of them — for example the Errata say:

Do not provide a sitemap or table of contents unless the site cannot be understood or navigated without it.WCAG Samurai Errata

Which to me seems rather odd. Why not? Why can’t you include a site map, if it is presented in such a way where it can be easily found by people who want to see it but isn’t ‘in the way’? What possible accessibility problem is posed by sensible inclusion of a site map?

Nevertheless, while I don’t entirely agree with all of the Samurai Errata — and indeed nor does Gian Sampson Wild herself, having written a very worthy technical review of the WCAG Samurai Errata — I do whole heartedly agree that the Samurai Errata is a useful resource for developers.

In my opinion however — and seemingly that of Patrick Lauke — the Samurai Errata are even weaker as regards cognitive disability than the original WCAG 1.0 checkpoints, striking out many criteria that would have been of benefit to those with cognitive disability — such as “supplement text with graphic or auditory presentations where they will facilitate comprehension of a page” and “create a style of presentation which that is consitent across pages” — both of these issues picked up by Gian herself in her review.

It’s fair to note however that the Errata themselves are aware of the issues around cognitive disabilities, specifically stating:

These errata do not substantively correct WCAG 1’s provisions for cognitive disabilities.WCAG Samurai Errata

…although perhaps not mentioning that they have in fact weakened the existing provision for cognitive disability.

The WCAG Samurai and WCAG Working Group: An Ironic Parallel

The Errata are also somewhat ‘imposed from on high’ by Joe and his secret buddies. That doesn’t necessarily mean that there’s anything wrong with them because of this, but it does mean that those behind the Errata need to be particularly careful responding to criticisms of the Errata.

If you are going to criiticise a group or individual for not listening to contrary opinions and not riding roughshod over people that disagree — as the Errata team contend is commonplace in the WCAG Working Group — then you’d damn well better be prepared to show the same attitude to people who disagree with you, otherwise you just come over as a whiny loser with double standards.

I’m going to be explicit here: I’m not saying that Joe, or any other member of the Errata team (assuming of course that it wasn’t just him!) are whiny losers with double standards. What I am saying is that they have to be prepared to take criticism on the chin and, even if they feel it is inaccurate or unwarranted, to give a considered and measured response to people who feel that they have raised a valid criticism and not to be rude or offensive in so doing.

Sharp comments directed at any dissenters would strike me as being ironically rather similar to the bullying behaviour that they criticise the Working Group for exhibiting. You can’t have it both ways.

Remember — whether you’re inside the working group looking out, or outside looking in, attack the argument, not the person.

And that brings us back to the concept of testability.

Testability: My Two Penn’orth

I work in the public sector. In my particular area, I’m probably seen as ‘the accessibility guy’. I know quite a bit about accessibility. I read about disability issues; I read blogs and websites that relate to accessibility issues. I read and contribute to AccessifyForum. I have extensive experience in trying — as best as I can — to evaluate the accessibility of websites.

So what generally happens when I’m talking to someone about WCAG 1.0? Generally the person I’m talking to — whether service user comissioning a website, or fellow developer comparing their site against our standards — wants to know whether or not their site complies with the appropriate WCAG 1.0 checkpoints.

They don’t particularly want to know how someone with disability X is going to come along and use their website; they don’t want to know about the assistive technologies associated with disability X; it’s incredibly unlikely that they even care about the social and medical models of disability. What they want to do is test whether or not their site is accessible.

Now we know it’s impossible to create the “Philosopher’s Site”: this theoretical mystical site that can be accessed and used by everyone, irrespective of their disability. But what we tend to do is to try and achieve a particular Conformance level that fits our need and the need of our visitors.

And that’s all the users need to know. Not “what techniques have been used to make my site accessible”; “what disabilities will this impact upon” or anything like that. What they want to know is — can I test my site and show that it is accessible.

If you keep testability in there, then yes you can. Importantly, it also means that web developers who for one perfectly sensible reason or another didn’t decide to focus around accessibility can reference the guidelines and test their sites for conformance without needing to know the guidelines inside-out.

Yes, I know it would be ideal if everyone was forced to understand the concepts of accessibility, disability and assistive technology before they were allowed near a computer, but that isn’t going to happen. Making accessibility more approachable and more understandable to those who previously haven’t had an interest in it is surely a good thing.

Of equal importance is the idea that testability removes accessibility evaluation from the geeks, nerds and social illiterates (or “web designers” as I believe we’re sometimes termed) and means that, since the criteria are user-centric, many of them (i.e. the ones that aren’t too technical) can be tested by pretty much anyone.

You don’t have to be a web designer with an intricate knowledge of HTML semantics to be able to determine whether or not a time-out setting is adjustable. You don’t need a degree in accessible web design to test whether or not your site can be accessed via the keyboard.

Admittedly, a lot of the guidelines aren’t as straightfoward as this but the fact that we can much more easily demonstrate accessibility conformance to people who maybe knew bugger-all about the issues to begin with is again surely a good thing?

And for that reason, after a bit of thought, I’ve now deviated from my initial response to Gian’s article, where I said:

What about something that says checkpoints (oh, all right, “success crtieria”) must be testable wherever possible? That way success criteria relating to cognitive disabilities etc can be included but testability is still brought into the equation wherever it can be?JackP

Now I’ve had a chance to think about it a bit more, I think the concept of testability is just too important to lose.

That’s not to stop other people from going further — I’ll still try to incorporate what I believe to be good practice even if I can’t test it — but what it does do is to lower the entry ‘barrier’: you would no longer need to understand concepts of disability and of assistive technology in order to be able to test your sites.

For that reason, I’d like to respectfully disagree with Gian, and say that while I acknowledge that the testability concept does come at a cost as she rightfully points out, I think that cost is worth paying. I do thank her for her well thought out and passionate argument, and I hope that she does not feel that I have been critical of her personally simply because I don’t agree with all of her argument, as that was never my intention.

Which brings me to my postscript.

PS: Isn’t About Time We Stopped All This Bitching And Got On With Making Sites Accessible?

I think the accessibility community has become too tainted by bitchiness, tetchiness, sniping and back-swiping, so it’s about time we all agreed to be pleasant and nice to one another even where we disagree. I’m also tempted to introduce a formal BAD COP, to see if we can get accessibility bloggers to agree to sign up to and be civil and polite to each other, even when disagreeing with someone else’s point of view.

I believe that this can also be referred to as “acting like grown-ups”.

If anyone thinks any kind of formal policy for arguments would actually help, I’m willing to have a go at drafting one (while of course being fully aware of the fact that everyone will disagree with my points of view!). Anyone think it’s worth a try?