Accessibility Testing – Evaluation of Tools

Posted by Albert Gareev on Nov 24, 2014 | Categories: AccessibilityReviewsTools

For any project, it’s typical to conduct assessment and evaluation of tools before acquiring licenses and putting them into use.

Below I’m sharing evaluation matrix composed based on my review of accessibility requirements.



Checking By Tool

Review by person

1.1 Text Alternatives

Provide text alternatives for any non-text content
1.1.1 (A) – Alternative Text All images, form image buttons, and image map hot spots have appropriate, equivalent alternative text. Partial Detailed
1.1.1 (A) – Alterative Text – non-content images Images that do not convey content, are decorative, or contain content that is already conveyed in text are given null alt text (alt=””) or implemented as CSS backgrounds. Full Brief
All linked images have descriptive alternative text. Partial Detailed
1.1.1 (A) – Alterative Text – complex images Equivalent alternatives to complex images are provided in context or on a separate (linked and/or referenced via longdesc) page Partial Detailed
1.1.1 (A) – Form buttons Form buttons have a descriptive value Partial Brief
1.1.1 (A) – Form inputs Form inputs have associated text labels or, if labels cannot be used, a descriptive title attribute Partial Brief
1.1.1 (A) – Embedded multimedia Embedded multimedia is identified via accessible text Partial Brief
1.1.1 (A) – Frames Frames are appropriately titled Partial Brief

1.2 Time-based Media

Provide alternatives for time-based media.  NOTE: If the audio or video is designated as an alternative to web content (e.g., an audio or sign language version of a web page, for example), then the web content itself serves as the alternative.
1.2.1 (A) Prerecorded Audio-only A descriptive text transcript (including all relevant visual and auditory clues and indicators) is provided for non-live, web-based audio (audio podcasts, MP3 files, etc.). N/A Detailed
1.2.1 (A) Prerecorded Video-only A text or audio description is provided for non-live, web-based video-only (e.g., video that has no audio track). N/A Detailed
1.2.2 (A) Captions (Prerecorded) Synchronized captions are provided for non-live, web-based video (YouTube videos, etc.) N/A Detailed
1.2.3 (A) Audio Description or Media Alternative (Prerecorded) A descriptive text transcript OR audio description audio track is provided for non-live, web-based video N/A Detailed
1.2.4 (AA) Captions (Live) Synchronized captions are provided for all live multimedia that contains audio (audio-only broadcasts, web casts, video conferences, Flash animations, etc.) N/A Detailed
1.2.5 (AA) Audio Description (Prerecorded) Audio descriptions are provided for all video content NOTE: Only required if the video conveys content visually that is not available in the default audio track. N/A Detailed

1.3 Adaptable

Create content that can be presented in different ways (for example simpler layout) without losing information or structure
1.3.1 (A) – Semantic Markup Semantic markup is used to designate headings (<h1>), lists (<ul>, <ol>, and <dl>), emphasized or special text (<strong>, <code>, <abbr>, <blockquote>, for example), etc. Semantic markup is used appropriately Partial Detailed
1.3.1 (A) – Tabular data Tables are used for tabular data. Where necessary, data cells are associated with their headers. Data table captions and summaries are used where appropriate Full Detailed
1.3.1 (A) – Related form elements Related form elements are grouped with fieldset/legend Full Brief
1.3.2 (A) – Meaningful Sequence The reading and navigation order (determined by code order) is logical and intuitive N/A Detailed
1.3.3 (A) – Instructions – visual Instructions do not rely upon shape, size, or visual location (e.g., “Click the square icon to continue” or “Instructions are in the right-hand column”). Partial Detailed
1.3.3 (A) – Instructions – auditory Instructions do not rely upon sound (e.g., “A beeping sound indicates you may continue.”). N/A Brief

1.4 Distinguishable

Make it easier for users to see and hear content including separating foreground from background
1.4.1 (A) – Colour Color is not used as the sole method of conveying content or distinguishing visual elements. Special Special
1.4.1 (A) – Colour – links Color alone is not used to distinguish links from surrounding text unless the luminance contrast between the link and the surrounding text is at least 3:1 and an additional differentiation (e.g., it becomes underlined) is provided when the link is hovered over or receives focus. Special Special
1.4.2 (A) – Audio Control A mechanism is provided to stop, pause, mute, or adjust volume for audio that automatically plays on a page for more than 3 seconds. Special Special
1.4.3 (AA) – Contrast ratio Text and images of text have a contrast ratio of at least 4.5:1. Special Special
1.4.3 (AA) – Contrast ratio – large text Large text (over 18 point or 14 point bold) has a contrast ratio of at least 3:1 Special Special
1.4.4 (AA) – Resize text The page is readable and functional when the text size is doubled. Special Special
1.4.5 (AA) – Images of text If the same visual presentation can be made using text alone, an image is not used to present that text. Special Special

2.1 Keyboard Access

Make all functionality available from a keyboard
2.1.1 (A) – Keyboard acessible All page functionality is available using the keyboard, unless the functionality cannot be accomplished in any known way using a keyboard (e.g., free hand drawing). Partial Detailed
2.1.1 (A) – Shortcuts Page-specified shortcut keys and accesskeys (accesskey should typically be avoided) do not conflict with existing browser and screen reader shortcuts. Partial Detailed
2.1.2 (AA) – Keyboard focus Keyboard focus is never locked or trapped at one particular page element. The user can navigate to and from all navigable page elements using only a keyboard. Partial Detailed

2.2 Enough Time

Provide users enough time to read and use content
2.2.1 (A) – Timing Adjustable If a page or application has a time limit, the user is given options to turn off, adjust, or extend that time limit. This is not a requirement for real-time events (e.g., an auction), where the time limit is absolutely required, or if the time limit is longer than 20 hours. N/A Brief
2.2.2 (A) – Auto moving content Automatically moving, blinking, or scrolling content that lasts longer than 5 seconds can be paused, stopped, or hidden by the user. Moving, blinking, or scrolling can be used to draw attention to or highlight content as long as it lasts less than 5 seconds. N/A Special
2.2.2 (A) – Auto refresh content Automatically updating content (e.g., automatically redirecting or refreshing a page, a news ticker, AJAX updated field, a notification alert, etc.) can be paused, stopped, or hidden by the user or the user can manually control the timing of the updates. N/A Detailed
2.3 Seizures Do not design content in a way that is known to cause seizures
2.3.1 (A) – Content flashes No page content flashes more than 3 times per second unless that flashing content is sufficiently small and the flashes are of low contrast and do not contain too much red. (See general flash and red flash thresholds) Partial Special

2.4 Navigable

Provide ways to help users navigate, find content, and determine where they are
2.4.1 (A) – Bypass blocks A link is provided to skip navigation and other page elements that are repeated across web pages. If a page has a proper heading structure, this may be considered a sufficient technique instead of a “Skip to main content” link. Note that navigating by headings is not yet supported in all browsers. If a page uses frames and the frames are appropriately titled, this is a sufficient technique for bypassing individual frames. Partial Detailed
2.4.2 (A)- Page titles The web page has a descriptive and informative page title. Full Brief
2.4.3 (A) – Focus order The navigation order of links, form elements, etc. is logical and intuitive. Partial Brief
2.4.4 (A) – Link Purpose The purpose of each link (or form image button or image map hotspot) can be determined from the link text alone, or from the link text and it’s context (e.g., surrounding paragraph, list item, table cell, or table headers). N/A Brief
2.4.4 (A) – Link Purpose – same text Links (or form image buttons) with the same text that go to different locations are readily distinguishable. Partial Brief
2.4.5 (AA) – Multiple ways Multiple ways are available to find other web pages on the site – at least two of: a list of related pages, table of contents, site map, site search, or list of all available web pages. N/A Brief
2.4.6 (AA) – Headings and Labels Page headings and labels for form and interactive controls are informative. Avoid duplicating heading (e.g., “More Details”) or label text (e.g., “First Name”) unless the structure provides adequate differentiation between them N/A Brief
2.4.7 (AA) – Focus visible It is visually apparent which page element has the current keyboard focus (i.e., as you tab through the page, you can see where you are). N/A Brief

3.1 Readable

Make text content readable and understandable
3.1.1 (A) – Language of Page The language of the page is identified using the HTML lang attribute (<html lang=”en”>, for example). Full Brief
3.1.2 (AA) – Language of Parts The language of page content that is in a different language is identified using the lang attribute (e.g., <blockquote lang=”es”>). Full Brief
3.2 Predictable Make Web pages appear and operate in predictable ways
3.2.1 (A) – On Focus When a page element receives focus, it does not result in a substantial change to the page, the spawning of a pop-up window, an additional change of keyboard focus, or any other change that could confuse or disorient the user. Partial Brief
3.2.2 (A) – On Input When a user inputs information or interacts with a control, it does not result in a substantial change to the page, the spawning of a pop-up window, an additional change of keyboard focus, or any other change that could confuse or disorient the user unless the user is informed of the change ahead of time. Partial Detailed
3.2.3 (AA) – Consistent Navigation Navigation links that are repeated on web pages do not change order when navigating through the site. N/A Detailed
3.2.4 (AA) – Consistent Identification Elements that have the same functionality across multiple web pages are consistently identified. For example, a search box at the top of the site should always be labeled the same way. N/A Detailed

3.3 Input Assistance

Help users avoid and correct mistakes
3.3.1 (A) – Error Identification Required form elements or form elements that require a specific format, value, or length provide this information within the element’s label (or if a label is not provided, within the element’s title attribute). N/A Detailed
3.3.1 (A) – Form Validation Errors If utilized, form validation errors are presented in an efficient, intuitive, and accessible manner. The error is clearly identified, quick access to the problematic element is provided, and user is allowed to easily fix the error and resubmit the form. N/A Detailed
3.3.2 (A) – Labels or Instructions Sufficient labels, cues, and instructions for required interactive elements are provided via instructions, examples, properly positioned form labels, and/or fieldsets/legends. N/A Detailed
3.3.3 (AA) – Error suggestion If an input error is detected (via client-side or server-side validation), provide suggestions for fixing the input in a timely and accessible manner. N/A Detailed
3.3.4 (AA) – Error prevention (Legal, Financial, Data) If the user can change or delete legal, financial, or test data, the changes/deletions can be reversed, verified, or confirmed. N/A Detailed

4.1 Robust

Maximize compatibility with current and future user agents, including assistive technologies
4.1.1 (A) – Parsing Significant HTML/XHTML validation/parsing errors are avoided. Check at Full Detailed
4.1.2 (A) – Name, Role, Value Markup is used in a way that facilitates accessibility. This includes following the HTML/XHTML specifications and using forms, form labels, frame titles, etc. appropriately. Full Detailed

Browser Compatibility

Internet Explorer compatibility back to version 6.0
Desktop browser compatibility in Firefox, Chrome, Safari and Opera
Mobile browser compatibility in iPhone, iPad, Android and Blackberry

Broken Links and Errors

Broken links and unexpected HTTP error codes
Missing images and script errors
Invalid SSL certificates, missing DNS entries and expired domains
Badly configured RSS feeds and 404 handlers
Parsing errors

Spell and Content Checking

Find blank pages and placeholder text such as “Lorem Ipsum” and “TODO”
Custom dictionary for unusual words like product names

Web Standards

Validate HTML and XHTML
Validate CSS

Privacy Standards

Missing privacy statements
Setting cookies or using tracking images without a privacy statement

Supported document formats

HTML*, XHTML*, CSS, JavaScript
Flash SWF

Validate an entire website in one go

Select exactly what parts of your site to validate and what to avoid
Select exactly what validations to perform, by site, by page and by item
Validate pages behind one or more forms, such as login forms


I also added “collateral value” categories typically offered by accessibility checking tools, and closely related to Accessibility Testing.

  • One response to "Accessibility Testing – Evaluation of Tools"

  • @halperinko - Kobi Halperin
    26th November 2014 at 10:10

    Very useful list!
    For all products – I wouldn’t waste time visually testing for spelling – but rather recommend making use of the Locale files, and MS-Word and other tools holding spellers to find most failures at once.
    Then in case someone did not use the locale file – we may find residues during normal testing – but at least we will waste much less time on seeking, reporting each item and manipulating vast number of minor bugs.
    (If you do find such “hard-coded” bugs – I will leave it to you on how to handle that “developer” – Hammers, bats, all options are open :-) )

    [Albert’s comment: not sure I follow. The list is from WCAG. My component is the assessment on what’s checkable by tools and to what extent. Spelling is not really a problem, not in terms of Web Accessibility.]

Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported
This work by Albert Gareev is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported.