Category: Testing and Evaluation

Conformance Testing

Also known as: Compliance Testing, Type Testing

Simply put

Conformance testing is a process used to determine whether a product, system, or service meets the requirements set out in a specified standard. In digital accessibility, this generally means checking how well a website or application follows the criteria of a recognized standard such as the Web Content Accessibility Guidelines (WCAG). It is important to note that meeting a standard's criteria does not by itself guarantee a fully accessible experience for all users or immunity from legal claims.

Formal definition

Conformance testing is the process of evaluating a single implementation of a specified standard to determine the extent to which it faithfully meets that standard's individual requirements. It involves capturing the technical description of a specification and measuring an implementation against it. In an accessibility context, conformance testing commonly assesses an implementation against WCAG success criteria at a targeted conformance level (most often Level AA). Because automated tools detect only a portion of possible issues, thorough conformance evaluation generally requires a combination of automated checks, manual review, and assistive technology testing. Conformance to a technical standard is distinct from usability and from legal compliance, which is governed separately through applicable regulation and case law; this entry is not legal advice, and readers should consult qualified counsel regarding specific obligations.

Why it matters

Conformance testing gives organizations a structured, repeatable way to measure how well a digital product aligns with a recognized standard such as WCAG. Without a defined process for measuring an implementation against specific requirements, accessibility claims tend to rest on assumptions rather than evidence. A documented conformance evaluation, targeted at a stated conformance level (most commonly WCAG Level AA), helps teams identify concrete gaps, prioritize remediation, and communicate progress to stakeholders in terms tied to an external benchmark rather than internal opinion.

At the same time, it is important to be clear about what conformance testing does and does not establish. Meeting a standard's success criteria demonstrates conformance to that technical standard; it does not by itself guarantee a fully accessible experience for every user, nor does it guarantee immunity from legal claims. Legal compliance in the United States is governed separately through applicable regulation and case law, and it varies by the authority involved. Conformance results are best treated as strong supporting evidence of an accessibility effort, not as a definitive legal conclusion.

Because automated tools detect only a portion of possible issues, treating an automated pass as proof of conformance can create a false sense of security. Thorough conformance evaluation generally combines automated checks with manual review and assistive technology testing. Organizations that understand these limits are better positioned to invest testing resources where they matter and to avoid overstating the assurances that a conformance result can provide. This entry is not legal advice; readers should consult qualified counsel regarding specific obligations.

Who it's relevant to

Accessibility Engineers and QA Testers

These practitioners perform the hands-on work of measuring an implementation against WCAG success criteria at a targeted conformance level. They combine automated checks, manual review, and assistive technology testing, understanding that automated tools alone detect only a portion of possible issues.

Compliance Officers

Compliance officers use conformance testing results as documented evidence of how well a product aligns with a recognized standard. They should recognize that conformance to a technical standard is distinct from legal compliance, which is governed separately through applicable regulation and case law.

UX Designers

Designers benefit from understanding that meeting a standard's criteria does not by itself guarantee a fully accessible or usable experience for all users. Conformance testing highlights measurable gaps, but usability considerations extend beyond what conformance alone captures.

Legal Counsel

Counsel may reference conformance testing results when assessing an organization's accessibility posture, while keeping in mind that conformance is not equivalent to legal compliance and that obligations vary by jurisdiction and evolving case law. This entry is not legal advice, and specific obligations should be evaluated by qualified counsel.

Business Leaders

Leaders rely on conformance testing to gauge progress against an external benchmark and to inform investment in remediation. They should treat conformance results as supporting evidence of an accessibility effort rather than as a guarantee of full accessibility or legal safety.

Inside Conformance Testing

Automated Testing

The use of software tools to scan digital content for detectable accessibility issues against WCAG success criteria. Automated testing efficiently identifies certain machine-detectable problems but generally catches only a portion of potential issues and cannot evaluate context-dependent requirements.

Manual Testing

Human evaluation of accessibility features that automated tools cannot reliably assess, such as logical reading order, meaningful alternative text, keyboard operability, and appropriate use of headings and landmarks. Manual review is generally required to complement automated results.

Assistive Technology Testing

Evaluation using tools such as screen readers, screen magnifiers, and voice control software to confirm how content behaves for people who rely on them. This helps surface usability problems that may not appear in either automated or standard manual review.

WCAG Success Criteria and Conformance Levels

Conformance testing is commonly measured against the Web Content Accessibility Guidelines published by the W3C, evaluated at levels A, AA, or AAA. Level AA is the target most frequently cited. Testers should confirm which WCAG version (2.0, 2.1, or 2.2) is being applied, as criteria differ across versions.

Scope and Sampling

A defined set of pages, screens, templates, or user flows selected for evaluation, along with the technical environments (browsers, devices, assistive technologies) tested. Clearly documenting scope helps clarify what conformance claims cover and what remains untested.

Documentation of Results

A record of findings, including which criteria were evaluated, methods used, issues identified, and their severity. Structured documentation supports remediation planning and may inform conformance statements such as an Accessibility Conformance Report.

Common questions

Answers to the questions practitioners most commonly ask about Conformance Testing.

Does passing conformance testing mean my website is legally compliant with the ADA?

No. Conformance testing measures whether content meets specified WCAG success criteria; it does not by itself guarantee compliance with the ADA or immunity from legal claims. The ADA is a US civil rights law that does not contain detailed web technical specifications, though courts and settlements have often referenced WCAG as a benchmark. Whether a particular level of conformance satisfies a legal obligation depends on the applicable authority, jurisdiction, and evolving case law and regulation. This is not legal advice, and you should consult qualified legal counsel regarding your specific obligations.

If automated testing tools report no errors, does that mean my content is accessible?

Not necessarily. Automated tools detect only a portion of potential accessibility issues, and a clean automated report does not confirm that content is usable by all people with disabilities. Many success criteria require human judgment to evaluate, so manual testing and testing with assistive technologies are generally needed alongside automated checks. Conformance to success criteria also does not by itself guarantee a good usability experience for every user.

Which WCAG version and conformance level should conformance testing target?

The choice depends on your goals and any applicable requirements. WCAG is published in versions including 2.0, 2.1, and 2.2, each with A, AA, and AAA levels. AA is the level most commonly cited as a target. Later versions add success criteria not present in earlier ones, so it is important to identify which version and level you are testing against and to confirm any requirements tied to a specific regulation, agency guidance, or settlement rather than assuming a single universal standard applies.

How should automated and manual testing be combined in a conformance testing process?

Because automated tools cover only part of the applicable success criteria, a testing process commonly pairs automated scanning with manual evaluation and testing using assistive technologies. Automated checks can efficiently surface certain machine-detectable issues, while manual review addresses criteria that require human judgment, such as evaluating meaningful sequence, context, or the usability of interactions. Documenting which criteria were assessed by which method helps clarify the scope and limits of the results.

What should conformance testing documentation include?

Documentation generally records which WCAG version and conformance level were used as the benchmark, the scope of pages or components tested, the testing methods applied (automated, manual, and assistive technology), and the results against individual success criteria. Noting what was in and out of scope helps readers understand the limits of the findings. Because conformance results reflect a point in time, records may also indicate the date of testing and the configurations evaluated.

How often should conformance testing be performed?

Conformance reflects the state of content at the time it is tested, so results can become outdated as content, code, and third-party components change. Testing is commonly repeated when significant changes are made and on a recurring basis as part of ongoing quality processes. Requirements and guidelines also evolve through regulation and case law, so periodic reassessment against the current applicable standard is generally advisable.

Common misconceptions

Passing an automated accessibility scan means content is fully accessible and WCAG conformant.

Automated testing detects only a portion of potential accessibility issues. Many WCAG success criteria require human judgment, so manual and assistive technology testing are generally necessary to assess conformance meaningfully.

Meeting WCAG success criteria guarantees an accessible experience for all users and immunity from legal claims.

Conformance with technical criteria does not guarantee usability for every user, nor does it ensure protection from legal claims. Conformance, usability, and legal compliance are related but distinct, and legal requirements evolve through regulation and case law. This is not legal advice; consult qualified counsel.

Conformance testing to a WCAG version is a one-time task.

Digital content, WCAG versions, and applicable regulations change over time. Conformance reflects the state of tested content at a point in time and within a defined scope, so periodic retesting is generally advisable as content and standards evolve.

Best practices

Combine automated, manual, and assistive technology testing rather than relying on any single method, since each detects different categories of issues.

Explicitly state the WCAG version (2.0, 2.1, or 2.2) and conformance level (commonly AA) being tested against, and confirm which success criteria apply to that version and level.

Define and document the scope of testing, including the pages, user flows, browsers, devices, and assistive technologies evaluated, so conformance claims accurately reflect what was tested.

Record findings in structured documentation that identifies the criteria evaluated, methods used, and severity of issues to support remediation and any conformance statements.

Treat conformance testing as an ongoing process, retesting when content changes or when applicable WCAG versions or regulations are updated.

Recognize that conformance testing addresses technical criteria and usability indicators but is not a substitute for legal advice; consult qualified counsel and monitor current agency rulemaking for compliance questions.