Is Accessibility Measurement Harmful?

Monday, March 10, 2008 0:37 | Filed in Accessibility, Public Sector, Standards

This is the question I’ve been asking myself for some time. Note specific use of the word measurement. I believe accessibility assessments are useful; but I’m not convinced that trying to assign a rating to a site is at all helpful and indeed I think it can actually be harmful.

I’ve been looking at the accessibility of some LA sites with other PSWMG members as part of an accessibility supplement to the Better Connected 2008 report, so I need to be clear here. I’m not speaking on behalf of PSWMG, I’m not speaking on behalf of anyone other than myself here.

These are simply my thoughts and opinions, which may or may not be shared by others, and to be honest haven’t changed much over the last year or so. This post may have been somewhat inspired by this year’s SOCITM kerfuffle, and the fact that I feel as interim chair of the PSWMG, I should explain my personal viewpoint.

Again, to clarify, the combined viewpoint of those PSWMG members tasked with producing the accessibility details for SOCITM will be covered as part of SOCITM’s accessibility supplement, so I can’t cover them here, and I can’t claim to know the viewpoint of all the PSWMG members (like any organisation, I’m sure some people would disagree with others!), so this is just my take on things. Although reading the Better Connected discussion on PSF, it is clear many have a similar viewpoint to myself…

Are we sufficiently clear on that? Good. Then lets begin.

Firstly, the current kerfuffle.

It is with the utmost regret we find ourselves having to report, yet again, Socitm Insight – in collusion with RNIB — has delivered a massive betrayal, dis-service and kick in the teeth over website accessibility to council website managers and developers the length and breadth of the land.

…an ‘important amendment’ to the original statement claiming, unbelievably, that 92% of council websites are inaccessible. This, we maintain, is a nonsense, for reasons that will be made clear and (just as we said last year) is a matter which brings the entire Report into disrepute.

The RNIB ‘rankings’ give no true picture of the current state of local authority website accessibility and should therefore be ignored completely.

Public Sector Forums

Strong stuff, no?

So why such strong language, exactly?

I think mostly because this problem cropped up last year too. Last year, the methodology was similar — testing against WCAG 1.0 to ‘measure’ the accessibility of sites. Last year it was criticised:

BC’s assessment of the accessibility of local authority websites is fundamentally flawed. Admittedly this is a reflection of the use of the Web Content Accessibility Guidelines 1.0 as the instrument of measurement…

The solution, however unpalatable it might be to the bean counters who seem to have a desperate need to rank and score us all, is to abandon the concept of ranking 468 websites for accessibility, and to stop testing them against an 8 year-old set of guidelines.

…The current state of affairs is like asking the Michelin Guide to judge restaurants on the quality of their cutlery

Dan Champion on Blether

That was from 2007. So were my comments:

So maybe it’s time to stop “ranking” websites in terms of accessibility? Some webistes which failed to achieve level AA conformace might actually have been more accessible to users with disabilities than some of the ones that did achieve AA conformance.

Let’s stop saying that WCAG is a measure of accessibility. It isn’t. WCAG shows you things you can do that will likely make your site more accessible. But throw some real people at your site and it either will or it won’t be accessible to them, irrespective of what your WCAG conformance level says.

Remember: WCAG Conformance is not the same as accessibility.

Real people. Real tests. Real progress?

Me, on Assessing Accessibility Part 1:The SOCITM Story

So both Dan and myself (and others) have made our positions clear. WCAG Conformance is not the same as accessibility. We were quite clear last year that this was our stance.

Unfortunately, despite the fact that many people seemed to agree with us, no-one seems to have listened

And the worry I have is that by continuing to grade sites against WCAG 1.0, people will (quite reasonably, but incorrectly) assume that a site which passes WCAG 1.0 at Level A will be more accessible to users than one which doesn’t, and that a site which passes WCAG 1.0 at level AA will be more accessible than one which doesn’t, and so on. This is nonsense.

All WCAG 1.0 conformance can tell you is to what extent the site complies with WCAG. Nothing more, and it certainly can’t give you an accurate rating of the ‘accessibility’ of a site.

Now I’ve not read Better Connected 2008. I’ve only been involved in the accessibility supplement to be released later. It may be that SOCITM do not imply WCAG Conformance equates to real-world accessibility. But unless they specifically state that it doesn’t, people will assume that because that’s what they are measuring, that it’s somehow worthwhile. And it isn’t.

Here’s why.

It is not possible to mark up changes in language in an alt text.

If you don’t mark up changes in language, you can’t reach the most basic WCAG 1.0 conformance level. Therefore if you have a picture which shows some text in more than one language (e.g. something with “Welcome” and “Bonjour!” on it), you cannot conform to even the most basic level of WCAG 1.0.

That doesn’t mean your site is inaccessible. In this case — because you only need one fail, anywhere on your site to stop you reaching a conformance level — all this would mean was that screen reader users would probably have Bonjour pronounced to them incorrectly. And if their screen reader didn’t come with a French language pack in the first place, it would be pronounced incorrectly anyway.

Does that mean :

…one or more groups will find it impossible to access information in the document.WCAG 1.0

Of course it doesn’t.

And, in case you are wondering whether there are any real world situations where this sort of thing occurs, yes. Many sites frequently use images of foreign text (Urdu, Gujarati, Cantonese etc) to provide assistance to communities who have that as a first language. If the alt text is not marked in the appropriate language, this is a WCAG 1.0 fail.

So the site has two choices:

  1. offer a poorer service, by not offering information in multiple languages and comply with WCAG 1.0
  2. fail WCAG 1.0 by offering images of multiple languages and have a site of more benefit to its users

The theoretical ideal is for the information to be presented as text in the appropriate language, but that isn’t always possible. You would require each site, each user of the site (irrespective of whether they’re accessing from home, internet cafe etc), each resident, each council PC to be installed with all of the language packs for all of these languages or it wouldn’t work. That’s just not practical. Using images is.

That is why a “measure” against WCAG 1.0 is plainly a nonsense. The site that provides the better service is seemingly penalised for doing so, and if SOCITM continue to take a an approach to accessibility based on guidelines (particularly out-dated guidelines) rather than real user experience, then they are doing their membership a disservice: they risk pushing down real-world accessibility at the expense of conforming to 9 year old guidelines.

The RNIB have done a good job of measuring how well sites conform to WCAG 1.0 for the Better Connected report. That’s what they were asked to do, and I don’t criticise them for it (I have a great deal of respect for their team). Unfortunately, that tells you nothing useful about the real-world accessibility of the sites in question. If the RNIB had not been asked to focus on WCAG 1.0 on ‘measure’ accessibility, I think they would have come out with something more useful.

I’d like to think that SOCITM will understand this and will look to improve the way they assess the accessibility of sites next year, and instead of using a measurement which (sometimes incorrectly) ranks sites against one another, I’d rather they took a user-centric approach, looking at common tasks and what problems people with different types of disabilities would face when carrying out these tasks. And I’d rather they didn’t use this as a “ranking” either, because just because a site does well in one area doesn’t mean it will necessarily do so in another. But then again, that’s what I was hoping for last year too…

But it just might be beneficial to list the most common accessibility problems encountered, the actual impact these would have on real-world users (for example, does the use of a deprecated attribute cause any problems for any users? If not, then why the hell fail a site for it as WCAG 1.0 would do?), and common ways which developers could resolve (or avoid) these problems.

That really could be used to help us all be “Better Connected”.

You can leave a response, or trackback from your own site.

19 Comments to Is Accessibility Measurement Harmful?

  1. Anthony says:

    March 10th, 2008 at 8:52 am

    Ooh don’t get me started on this SocITM rubbish…

    Even disregarding all the accessibility issues – which you have covered extremely well (as usual) – the whole thing just seems so subjective.

    We’ve been awarded the bottom one… Content or whatever it is this year… which is fair enough – until you look at the actual results.

    We actually picked up more points than a few sites that were awarded Transactional, never mind scoring three times scores in other areas than most of the other Content sites.

    The one thing that really annoyed us was that we were actually singled out for Best Practice for transactions!

    We seem to have missed out

    (a) because our server and CMS were having issues during the review (typical, we can’t complain really – but it’s so annoying!),
    (b) because our externally provided search system crapped out once during a search.

    That’s doubly annoying because we share the exact same search with all of the other districts in Cheshire plus the County… The County got a tick for good use of search but none of the districts did – despite us all using the exact same service! (We always advised against it in the first place….)

    The reviewers comments seemed to be full of very strange remarks too. I could go through them, but I’ve already written too much, I’ll just say they made some odd, very subjective observations, and it seems to have been based on these that we have missed out on Transactional.

    It’s just so annoying when you have bent over backwards to try and comply with what they want, only to be told it’s all wrong and not good enough.

    Especially as a few of our design decisions we made with their findings from last year in mind….


    Sorry for filling up the comments Jack, I’ll have a rant on my own blog when I get more details.

  2. says:

    August 31st, 2011 at 1:04 am

    How To Change Your Business…

    [...]It’s a known truth that right knowledge can be very important when we are doing something new and especially if it’s important to us.[...]…

  3. penny auction videos says:

    September 20th, 2011 at 9:25 am

    Penny Auction Blog…

    [...]We all know that right knowledge can be very important when we are doing something new and especially if it is something very important..[...]…

  4. phentermine 37.5 says:

    September 22nd, 2011 at 6:02 pm

    Phentermine Without Prescription Blog…

    [...]We are absolutely sure that right knowledge can be very important when doing work for the first time and especially if it’s important to us..[...]…

  5. youtube to mp3 converter says:

    September 23rd, 2011 at 10:12 am

    Making MP3 Of YouTube Songs…

    [...]When you have knowledge, skills and experience these are are crucial to make you happy at everything that you do.[...]…

  6. portrait paintings says:

    October 2nd, 2011 at 7:39 pm

    Portraits of Famous People…

    [...]To have many skills you can successfully do at a lot of jobs and doing almost no mistakes…[...]…

  7. picture into painting says:

    October 4th, 2011 at 11:28 am

    Last News In World – Posted Here…

    [...]When you have knowledge, skills and experience these are are crucial to make you happy at all things in life.[...]…

  8. plastic surgery internet marketing says:

    November 4th, 2011 at 11:45 am

    How To Make Plastic Surgery Marketing Work…

    [...]With the right knowledge you can be successful at at many more jobs and making no bad decisions…[...]…

  9. dining solutions direct says:

    November 8th, 2011 at 10:12 pm

    Everything You Have To Know About Dining Solutions…

    [...]This is a new website on this topic. You can see more about it on the site[...]…

  10. Date says:

    December 18th, 2011 at 9:03 pm

    What Are The Best Methods To Meet Ladies…

    [...]Writing about a topic like this is pretty tough. It’s good to understand some individuals nevertheless care about this[...]…

  11. lunettes tag heuer says:

    December 23rd, 2011 at 4:32 pm

    Find Great Tag Heuer Sunglasses…

    [...]This is really cool. It is fantastic to understand a lot of people still care about this[...]…

  12. Djani Stipanicev says:

    December 27th, 2011 at 5:46 pm

    How To Do Joint Ventures…

    [...]Writing about a topic like this is pretty tough. It’s good to understand some individuals nevertheless care about this[...]…

  13. challenge coins says:

    January 2nd, 2012 at 5:41 pm

    There Are Many Hobbies You Can Enjoy…

    [...]This is really cool. Thank you for writting this[...]…

  14. Atlanta Roofing says:

    January 11th, 2012 at 12:11 am

    How To Get Better Roofing Offers In Atlanta…

    [...]Thank you for posting this exciting info. I will verify your web-site within the future.[...]…

  15. says:

    February 9th, 2012 at 4:23 pm

    Do You Want To Find Fun Online…

    [...]This is really cool. Keep up with the great writing.[...]…

  16. India Tourism says:

    February 17th, 2012 at 7:13 pm

    India Travel Online Reservations…

    [...]It’s good that individuals nonetheless know a good deal about factor like that. It’s good to know a number of people nonetheless care about this[...]…

  17. ghost says:

    March 30th, 2012 at 10:18 pm

    Real Ghost Busters…

    [...]I see you understand a great deal about what you’re writing about. Interesting read. I’ve just had a different concept pop into my mind[...]…

  18. dallas pools says:

    April 15th, 2012 at 2:11 pm

    Pool Services Can Make A Difference…

    [...]Are you unquestionably positive this really is the suitable information. I head a thing else. You will find out extra about this topic[...]…

  19. Easter says:

    December 10th, 2012 at 10:26 am

    That’s really thinnkig at an impressive level

Leave a comment