Accessibility Testing – Evaluation of Tools
For any project, it’s typical to conduct assessment and evaluation of tools before acquiring licenses and putting them into use.
Below I’m sharing evaluation matrix composed based on my review of accessibility requirements.
Requirement |
Description |
Checking By Tool |
Review by person |
1.1 Text Alternatives |
Provide text alternatives for any non-text content | ||
| 1.1.1 (A) – Alternative Text | All images, form image buttons, and image map hot spots have appropriate, equivalent alternative text. | Partial | Detailed |
| 1.1.1 (A) – Alterative Text – non-content images | Images that do not convey content, are decorative, or contain content that is already conveyed in text are given null alt text (alt=””) or implemented as CSS backgrounds. | Full | Brief |
| All linked images have descriptive alternative text. | Partial | Detailed | |
| 1.1.1 (A) – Alterative Text – complex images | Equivalent alternatives to complex images are provided in context or on a separate (linked and/or referenced via longdesc) page | Partial | Detailed |
| 1.1.1 (A) – Form buttons | Form buttons have a descriptive value | Partial | Brief |
| 1.1.1 (A) – Form inputs | Form inputs have associated text labels or, if labels cannot be used, a descriptive title attribute | Partial | Brief |
| 1.1.1 (A) – Embedded multimedia | Embedded multimedia is identified via accessible text | Partial | Brief |
| 1.1.1 (A) – Frames | Frames are appropriately titled | Partial | Brief |
1.2 Time-based Media |
Provide alternatives for time-based media. NOTE: If the audio or video is designated as an alternative to web content (e.g., an audio or sign language version of a web page, for example), then the web content itself serves as the alternative. | ||
| 1.2.1 (A) Prerecorded Audio-only | A descriptive text transcript (including all relevant visual and auditory clues and indicators) is provided for non-live, web-based audio (audio podcasts, MP3 files, etc.). | N/A | Detailed |
| 1.2.1 (A) Prerecorded Video-only | A text or audio description is provided for non-live, web-based video-only (e.g., video that has no audio track). | N/A | Detailed |
| 1.2.2 (A) Captions (Prerecorded) | Synchronized captions are provided for non-live, web-based video (YouTube videos, etc.) | N/A | Detailed |
| 1.2.3 (A) Audio Description or Media Alternative (Prerecorded) | A descriptive text transcript OR audio description audio track is provided for non-live, web-based video | N/A | Detailed |
| 1.2.4 (AA) Captions (Live) | Synchronized captions are provided for all live multimedia that contains audio (audio-only broadcasts, web casts, video conferences, Flash animations, etc.) | N/A | Detailed |
| 1.2.5 (AA) Audio Description (Prerecorded) | Audio descriptions are provided for all video content NOTE: Only required if the video conveys content visually that is not available in the default audio track. | N/A | Detailed |
1.3 Adaptable |
Create content that can be presented in different ways (for example simpler layout) without losing information or structure | ||
| 1.3.1 (A) – Semantic Markup | Semantic markup is used to designate headings (<h1>), lists (<ul>, <ol>, and <dl>), emphasized or special text (<strong>, <code>, <abbr>, <blockquote>, for example), etc. Semantic markup is used appropriately | Partial | Detailed |
| 1.3.1 (A) – Tabular data | Tables are used for tabular data. Where necessary, data cells are associated with their headers. Data table captions and summaries are used where appropriate | Full | Detailed |
| 1.3.1 (A) – Related form elements | Related form elements are grouped with fieldset/legend | Full | Brief |
| 1.3.2 (A) – Meaningful Sequence | The reading and navigation order (determined by code order) is logical and intuitive | N/A | Detailed |
| 1.3.3 (A) – Instructions – visual | Instructions do not rely upon shape, size, or visual location (e.g., “Click the square icon to continue” or “Instructions are in the right-hand column”). | Partial | Detailed |
| 1.3.3 (A) – Instructions – auditory | Instructions do not rely upon sound (e.g., “A beeping sound indicates you may continue.”). | N/A | Brief |
1.4 Distinguishable |
Make it easier for users to see and hear content including separating foreground from background | ||
| 1.4.1 (A) – Colour | Color is not used as the sole method of conveying content or distinguishing visual elements. | Special | Special |
| 1.4.1 (A) – Colour – links | Color alone is not used to distinguish links from surrounding text unless the luminance contrast between the link and the surrounding text is at least 3:1 and an additional differentiation (e.g., it becomes underlined) is provided when the link is hovered over or receives focus. | Special | Special |
| 1.4.2 (A) – Audio Control | A mechanism is provided to stop, pause, mute, or adjust volume for audio that automatically plays on a page for more than 3 seconds. | Special | Special |
| 1.4.3 (AA) – Contrast ratio | Text and images of text have a contrast ratio of at least 4.5:1. | Special | Special |
| 1.4.3 (AA) – Contrast ratio – large text | Large text (over 18 point or 14 point bold) has a contrast ratio of at least 3:1 | Special | Special |
| 1.4.4 (AA) – Resize text | The page is readable and functional when the text size is doubled. | Special | Special |
| 1.4.5 (AA) – Images of text | If the same visual presentation can be made using text alone, an image is not used to present that text. | Special | Special |
2.1 Keyboard Access |
Make all functionality available from a keyboard | ||
| 2.1.1 (A) – Keyboard acessible | All page functionality is available using the keyboard, unless the functionality cannot be accomplished in any known way using a keyboard (e.g., free hand drawing). | Partial | Detailed |
| 2.1.1 (A) – Shortcuts | Page-specified shortcut keys and accesskeys (accesskey should typically be avoided) do not conflict with existing browser and screen reader shortcuts. | Partial | Detailed |
| 2.1.2 (AA) – Keyboard focus | Keyboard focus is never locked or trapped at one particular page element. The user can navigate to and from all navigable page elements using only a keyboard. | Partial | Detailed |
2.2 Enough Time |
Provide users enough time to read and use content | ||
| 2.2.1 (A) – Timing Adjustable | If a page or application has a time limit, the user is given options to turn off, adjust, or extend that time limit. This is not a requirement for real-time events (e.g., an auction), where the time limit is absolutely required, or if the time limit is longer than 20 hours. | N/A | Brief |
| 2.2.2 (A) – Auto moving content | Automatically moving, blinking, or scrolling content that lasts longer than 5 seconds can be paused, stopped, or hidden by the user. Moving, blinking, or scrolling can be used to draw attention to or highlight content as long as it lasts less than 5 seconds. | N/A | Special |
| 2.2.2 (A) – Auto refresh content | Automatically updating content (e.g., automatically redirecting or refreshing a page, a news ticker, AJAX updated field, a notification alert, etc.) can be paused, stopped, or hidden by the user or the user can manually control the timing of the updates. | N/A | Detailed |
| 2.3 Seizures | Do not design content in a way that is known to cause seizures | ||
| 2.3.1 (A) – Content flashes | No page content flashes more than 3 times per second unless that flashing content is sufficiently small and the flashes are of low contrast and do not contain too much red. (See general flash and red flash thresholds) | Partial | Special |
2.4 Navigable |
Provide ways to help users navigate, find content, and determine where they are | ||
| 2.4.1 (A) – Bypass blocks | A link is provided to skip navigation and other page elements that are repeated across web pages. If a page has a proper heading structure, this may be considered a sufficient technique instead of a “Skip to main content” link. Note that navigating by headings is not yet supported in all browsers. If a page uses frames and the frames are appropriately titled, this is a sufficient technique for bypassing individual frames. | Partial | Detailed |
| 2.4.2 (A)- Page titles | The web page has a descriptive and informative page title. | Full | Brief |
| 2.4.3 (A) – Focus order | The navigation order of links, form elements, etc. is logical and intuitive. | Partial | Brief |
| 2.4.4 (A) – Link Purpose | The purpose of each link (or form image button or image map hotspot) can be determined from the link text alone, or from the link text and it’s context (e.g., surrounding paragraph, list item, table cell, or table headers). | N/A | Brief |
| 2.4.4 (A) – Link Purpose – same text | Links (or form image buttons) with the same text that go to different locations are readily distinguishable. | Partial | Brief |
| 2.4.5 (AA) – Multiple ways | Multiple ways are available to find other web pages on the site – at least two of: a list of related pages, table of contents, site map, site search, or list of all available web pages. | N/A | Brief |
| 2.4.6 (AA) – Headings and Labels | Page headings and labels for form and interactive controls are informative. Avoid duplicating heading (e.g., “More Details”) or label text (e.g., “First Name”) unless the structure provides adequate differentiation between them | N/A | Brief |
| 2.4.7 (AA) – Focus visible | It is visually apparent which page element has the current keyboard focus (i.e., as you tab through the page, you can see where you are). | N/A | Brief |
3.1 Readable |
Make text content readable and understandable | ||
| 3.1.1 (A) – Language of Page | The language of the page is identified using the HTML lang attribute (<html lang=”en”>, for example). | Full | Brief |
| 3.1.2 (AA) – Language of Parts | The language of page content that is in a different language is identified using the lang attribute (e.g., <blockquote lang=”es”>). | Full | Brief |
| 3.2 Predictable | Make Web pages appear and operate in predictable ways | ||
| 3.2.1 (A) – On Focus | When a page element receives focus, it does not result in a substantial change to the page, the spawning of a pop-up window, an additional change of keyboard focus, or any other change that could confuse or disorient the user. | Partial | Brief |
| 3.2.2 (A) – On Input | When a user inputs information or interacts with a control, it does not result in a substantial change to the page, the spawning of a pop-up window, an additional change of keyboard focus, or any other change that could confuse or disorient the user unless the user is informed of the change ahead of time. | Partial | Detailed |
| 3.2.3 (AA) – Consistent Navigation | Navigation links that are repeated on web pages do not change order when navigating through the site. | N/A | Detailed |
| 3.2.4 (AA) – Consistent Identification | Elements that have the same functionality across multiple web pages are consistently identified. For example, a search box at the top of the site should always be labeled the same way. | N/A | Detailed |
3.3 Input Assistance |
Help users avoid and correct mistakes | ||
| 3.3.1 (A) – Error Identification | Required form elements or form elements that require a specific format, value, or length provide this information within the element’s label (or if a label is not provided, within the element’s title attribute). | N/A | Detailed |
| 3.3.1 (A) – Form Validation Errors | If utilized, form validation errors are presented in an efficient, intuitive, and accessible manner. The error is clearly identified, quick access to the problematic element is provided, and user is allowed to easily fix the error and resubmit the form. | N/A | Detailed |
| 3.3.2 (A) – Labels or Instructions | Sufficient labels, cues, and instructions for required interactive elements are provided via instructions, examples, properly positioned form labels, and/or fieldsets/legends. | N/A | Detailed |
| 3.3.3 (AA) – Error suggestion | If an input error is detected (via client-side or server-side validation), provide suggestions for fixing the input in a timely and accessible manner. | N/A | Detailed |
| 3.3.4 (AA) – Error prevention (Legal, Financial, Data) | If the user can change or delete legal, financial, or test data, the changes/deletions can be reversed, verified, or confirmed. | N/A | Detailed |
4.1 Robust |
Maximize compatibility with current and future user agents, including assistive technologies | ||
| 4.1.1 (A) – Parsing | Significant HTML/XHTML validation/parsing errors are avoided. Check at http://validator.w3.org/ | Full | Detailed |
| 4.1.2 (A) – Name, Role, Value | Markup is used in a way that facilitates accessibility. This includes following the HTML/XHTML specifications and using forms, form labels, frame titles, etc. appropriately. | Full | Detailed |
Browser Compatibility |
|||
| Internet Explorer compatibility back to version 6.0 | |||
| Desktop browser compatibility in Firefox, Chrome, Safari and Opera | |||
| Mobile browser compatibility in iPhone, iPad, Android and Blackberry | |||
Broken Links and Errors |
|||
| Broken links and unexpected HTTP error codes | |||
| Missing images and script errors | |||
| Invalid SSL certificates, missing DNS entries and expired domains | |||
| Badly configured RSS feeds and 404 handlers | |||
| Parsing errors | |||
Spell and Content Checking |
|||
| English | |||
| French | |||
| Other | |||
| Find blank pages and placeholder text such as “Lorem Ipsum” and “TODO” | |||
| Custom dictionary for unusual words like product names | |||
Web Standards |
|||
| Validate HTML and XHTML | |||
| Validate CSS | |||
Privacy Standards |
|||
| Missing privacy statements | |||
| Setting cookies or using tracking images without a privacy statement | |||
Supported document formats |
|||
| HTML*, XHTML*, CSS, JavaScript | |||
| Flash SWF | |||
Validate an entire website in one go |
|||
| Select exactly what parts of your site to validate and what to avoid | |||
| Select exactly what validations to perform, by site, by page and by item | |||
| Validate pages behind one or more forms, such as login forms |
I also added “collateral value” categories typically offered by accessibility checking tools, and closely related to Accessibility Testing.
One response to "Accessibility Testing – Evaluation of Tools"
Very useful list!
For all products – I wouldn’t waste time visually testing for spelling – but rather recommend making use of the Locale files, and MS-Word and other tools holding spellers to find most failures at once.
Then in case someone did not use the locale file – we may find residues during normal testing – but at least we will waste much less time on seeking, reporting each item and manipulating vast number of minor bugs.
(If you do find such “hard-coded” bugs – I will leave it to you on how to handle that “developer” – Hammers, bats, all options are open :-) )
[Albert’s comment: not sure I follow. The list is from WCAG. My component is the assessment on what’s checkable by tools and to what extent. Spelling is not really a problem, not in terms of Web Accessibility.]