Website Accessibility Evaluation Methodology

Being an Accessibility Enthusiast, I generally get a question – How to evaluate accessibility of a website? Is there any approach or methodology which guarantees the accessible website? Well, I don’t think any methodology would guarantee a 100% accessible website until you get the website tested from real users – the users who face the pain of inaccessible websites.

Anyway, when the question became too common – I thought it's time to work on this and here I am presenting a methodology to evaluate accessible website. This methodology is evolved from fine tuning of industry’s best practices and Website Accessibility Conformance Evaluation Methodology by W3C.

This methodology includes the different phases of accessibility evaluation starting from collecting the baseline information to the last phase of retesting the fixed issues. This methodology can be used for existing websites and during earlier development stages of the website. This methodology can be used for evaluating static, dynamic, mobile and other types of websites.

Draw a Line
The objective of this phase is to collect the information and understand the accessibility requirement of the customer. This is accessibility evaluation preparation phase wherein tester collects prerequisite information such as target website, business nature, current accessibility status, identification of Assistive Technology and conformance standards to be implemented.

Let’s Explore
The tester explores the target website to get thorough understanding of website functionality, use and purpose. The exploration carried out in this phase helps in later stages when evaluation of target website takes place. The objective of this phase is to identify:
• Critical web pages of the website – The website is explored to identify mandatory functionality and critical web pages which are core of the website and might directly impact the business. The outcome of this step is the list of all critical web pages and mandatory functionality that user performs on the website.
• Common web pages of the website – The website is explored to identify the common web pages of the website. These web pages are generally directly linked to home page and main navigation of website and sometimes represent the different states of website. The outcome of this step is the list of all common web pages on the website.
• Dynamically generated web pages – The website might have web pages which are generated dynamically based on user input and important for the website. For example, product listing page on a website. The website is explored to identify such web pages on the website.
• Other relevant web pages – Here, the website is explored to identify the web pages that are more relevant to users with disabilities and accessibility of website such as web page informing about the website accessibility features, settings, shortcuts etc. The outcome of this step is list of such web pages.
• Variety of web page types – The target website may have web pages of variety of styles, templates, layout and structure, navigation and visual design etc. In this step, the website is explored to identify such web pages on the website. The outcome of this step is the list of all such web pages differentiated by varieties.

Collect the Samples
The objective of this step is to pick up the sample of all type of web pages which represent the target website. The purpose of this sampling procedure is to ensure the good enough accessibility evaluation of website with reasonable confidence. The sample set should include:
• Sample set of all critical web pages of the target website
• Sample set of Common web pages of the website
• Sample set of each dynamically generated web pages
• Sample set of Other Relevant web pages
• Sample set of each web page type
• Sample set representing the complete process
The sampling procedure could be skipped where it is feasible to evaluate all web pages in the target website; all the web pages are considered for accessibility procedure in such situation.
The selected sample sets are further evaluated to identify the common components such as navigation menu bars, headers and footers, search form etc. which do not require the re-evaluation on each occurrence.

Use Automation Tools
During this step, the selected sample set of web pages are evaluated by multiple automation tools as per the conformance standard selected in first step. The objective of automatic testing is to quickly identify the accessibility issues in the selected set of web pages of the website. The outcome of testing in this phase is:
• a list of confirmed accessibility issues which are failed to meet success criterion of selected conformance standard
• a list of potential accessibility risks which require manual intervention before next action
• a list of features which meets with the success criteria of selected conformance standard

Manual Testing with Assistive Technology
The tester further evaluates the selected set of web pages of target website manually to conformance with each success criterion of selected standard. The objective of this step is to identify the issues a disable user might be facing in using the target website. All content of the website is evaluated:
• to find out any occurrence of failure to meet success criteria of selected conformance standard
• to find out any occurrence of failure during the interaction of each web page and web page state along with complete process
• to find out any occurrence of failure in using the website with assistive technologies selected in first step
• to verify the list of potential accessibility risks generated in previous step
• to ensure that all content is supported by accessibility features

Make a Report, Retest
This step focuses on the analysis of reports collected from automated testing and observation made during manual testing process. The purpose is to identify all false positives and false negative before reporting the issues in the bug tracker. A report is produced with analysis outcome and recommendations and suggestions for all stakeholders.

Further, a decision should be made on retesting based on the outcome of the analysis. Retesting would have similar approach of testing - using automation tools followed by manual testing with AT.