Kilotest: Accessibility Testing Strategies

Why accessibility testing matters

About 5 minutes.

Accessibility is not a separate concern added to a finished product. It is an aspect of code quality, legal compliance, and basic usability — present or absent in every page you ship.

Code quality

Inaccessible code is often incorrect code. HTML specifications define semantic elements and attributes for good reasons, and violations frequently indicate structural errors that affect all users, not only users with disabilities. A missing autocomplete attribute, for example, is not a cosmetic gap — it tells the browser that the developer has not declared the purpose of a form field that users rely on.

Accessible code also tends to be more maintainable: clear structure, meaningful labels, and correct role assignments make code easier to understand and modify.

Legal compliance

In many jurisdictions, web accessibility is a legal requirement. The most widely referenced technical standard is the Web Content Accessibility Guidelines (WCAG), published by the W3C. WCAG 2.1 and 2.2 are referenced by law in the United States (Section 508, ADA), the European Union (European Accessibility Act), the United Kingdom (Equality Act), and many other countries.

Non-compliance exposes organisations to legal action, regulatory penalties, and reputational harm. Lawsuits and settlements involving web accessibility are documented each year in all of these jurisdictions.

Usability

Approximately 15–20% of people worldwide have a disability that can affect their use of the web. These include permanent conditions (blindness, deafness, motor impairments, cognitive disabilities) as well as temporary situations (a broken arm, recovering from eye surgery) and situational constraints (a noisy environment, a phone in bright sunlight, holding an infant).

Inaccessible code excludes or burdens these users. Features that are technically accessible also tend to benefit users without disabilities: clear labels reduce errors for everyone, logical keyboard flow speeds navigation, and high contrast aids readability in suboptimal lighting.

The three testing strategies

About 8 minutes.

Human testing

A person directly examines the page, using the browser and auxiliary tools to find accessibility issues. This is sometimes called manual testing, though the term can be misleading — modern human testing uses several tools; it is the human judgment that distinguishes this strategy.

Human testing typically involves three techniques:

• DOM inspection. The tester uses browser developer tools to examine the HTML structure, attributes, and the accessibility tree. The accessibility tree is the browser's representation of the page as assistive technologies see it — role, name, state, and value for each element.
• Keyboard navigation. The tester uses only the keyboard (Tab, Shift+Tab, Enter, Space, arrow keys) to navigate and operate the page. This reveals whether all interactive elements are reachable and operable without a mouse.
• Screen reader testing. The tester activates a screen reader — software that vocalises page content — and navigates the page by ear. Common screen readers include NVDA and JAWS on Windows, VoiceOver on macOS and iOS, and TalkBack on Android.

Effective human testing requires substantial knowledge: WCAG success criteria, ARIA authoring practices, HTML semantics, and proficiency with at least one screen reader. It also takes time — a thorough examination of a single page may take 30 minutes to several hours.

Rule-engine testing

Software applies a set of codified rules to a page and reports which rules are violated. This is sometimes called automated testing, though that label is also imprecise — what is automated is the rule application, not the judgment about what rules should exist.

Rule engines come in several forms:

• Browser extensions such as the axe DevTools extension and the WAVE Evaluation Tool run directly in the browser and report violations interactively.
• Installed engines such as Pa11y and Testaro run from the command line and can be integrated into development workflows.
• API services accept page URLs and return analysis results over the network.
• Ensemble services such as Kilotest run multiple rule engines against a page and consolidate the results. Because different engines encode different rules, an ensemble typically finds more issues than any single engine.

Rule-engine testing is fast — a full scan typically takes 1–5 minutes. It requires little accessibility knowledge from the user, since the rules encode the expertise. It can be run repeatedly, integrated into CI/CD pipelines, and applied to large numbers of pages.

AI-agent testing

An AI agent, as used in this tutorial, is a language model — software trained on large amounts of text to understand and generate language — configured with instructions to perform a specific task. The underlying model (such as Claude, GPT-4, or Gemini) is the system you access; the agent is the model combined with its instructions and the content you provide. For accessibility testing, the agent is given page content and asked to identify violations. This strategy is the newest of the three and is evolving rapidly.

The workflow involves three choices:

• Selecting a model. Different models vary in their knowledge of accessibility standards, their ability to reason about HTML structure, and their tendency to hallucinate. As of 2026, capable models include Claude (Anthropic), GPT-4 (OpenAI), and Gemini (Google).
• Designing instructions. The instructions — often called a prompt — specify what to test, what standards to apply, and how to report findings. Prompt quality significantly affects the quality of results.
• Providing page content. The AI agent must receive the relevant page content. This may be the raw HTML source, the rendered DOM, screenshots, or some combination. What you provide determines what the agent can find.

AI-agent testing can reason about issues that resist mechanical rules — for example, whether image alternative text is meaningful in context, or whether a form label is genuinely helpful. Its output is available within minutes and requires no specialist tools beyond access to an AI agent. However, AI agents can produce false findings and can miss real ones, and their results may change as the underlying models are updated.

Choosing among the strategies

About 12 minutes.

The three strategies are not alternatives — they are complements. A mature accessibility testing practice uses all three. Understanding their differences helps you decide how to allocate effort.

Time to results

Human testing

Slowest. A thorough examination of a single page takes 30 minutes to several hours, depending on the page's complexity and the scope of testing.

Rule-engine testing

Fastest. A full-page scan typically completes in 1–5 minutes. An ensemble service such as Kilotest may take somewhat longer.

AI-agent testing

Moderate. Depending on the scope of instructions, the amount of page content provided, and the AI agent's processing speed, 2–10 minutes is typical for focused testing.

Financial cost

Human testing

The cost of a skilled tester's time. Specialist accessibility consultants are expensive. Internal developers trained in accessibility are a recurring investment.

Rule-engine testing

Many rule engines are free and open-source. Commercial tools and API services typically charge per page or per month. Kilotest is currently free. Costs are generally low relative to the number of tests run.

AI-agent testing

Most AI services charge per token (unit of text) processed. As of 2026, asking an AI agent to check a single page against one WCAG success criterion typically costs between $0.01 and $0.10 depending on the model used. Prices are falling over time.

What can each strategy find?

Human testing

The broadest potential coverage. A skilled human can find any issue that is observable through the browser, developer tools, or a screen reader. Human testing is the only strategy that can reliably assess whether content is usable — not merely technically compliant — for users with disabilities. It is also the only strategy that can catch issues that only appear during interaction (filling out a multi-step form, for instance) or that depend on context (whether an image description is accurate given the surrounding text).

Rule-engine testing

Limited to issues for which explicit rules have been written. Rules can be precise and comprehensive for clearly defined requirements (for example: every <img> must have an alt attribute). They cannot assess subjective quality (for example: whether that alt text is actually helpful). Some issues are detectable in principle but not yet covered by available engines.

AI-agent testing

Broader than rule engines for issues requiring reasoning, narrower than humans for issues requiring interaction or lived experience. An AI agent can assess whether image alternative text is plausibly meaningful, whether instructions are clear, or whether form labels are adequate. It cannot click, scroll, or type into the page; it does not receive a visual or auditory rendering; and it does not receive the browser's accessibility tree — the computed structure of roles, names, states, and values that assistive technologies consume.

False positives (incorrect findings)

Human testing

Low, in the hands of a knowledgeable tester. A human who understands the standards can distinguish genuine violations from compliant edge cases. Inexperienced testers produce more false positives.

Rule-engine testing

Moderate and tool-dependent. Rules that cannot fully determine compliance from the code alone (for example: whether an image needs alternative text) may produce incorrect results. Different engines disagree about the same page, and some findings require human review to confirm.

AI-agent testing

Variable and model-dependent. Current models sometimes misidentify compliant patterns as violations, particularly when their knowledge of applicable standards is incomplete or when the HTML structure is complex. Critical review of AI output is essential.

False negatives (missed defects)

Human testing

Low for experienced testers working systematically, but not zero. Testers can overlook attributes on elements they do not specifically inspect. Issues that only manifest in edge cases or under specific conditions may be missed if those cases are not tested. Consistency across large page sets can be difficult to maintain.

Rule-engine testing

Any issue not covered by a rule will be missed. No current rule engine covers all WCAG success criteria. Engines also differ in which rules they implement: a single engine may miss issues that another engine catches. Ensemble testing reduces the false negative rate relative to any single engine.

AI-agent testing

Depends heavily on what content is provided to the AI agent. If the agent receives only the raw HTML source of a page whose forms are rendered by JavaScript, it will miss all form-related issues entirely. If the agent's knowledge of valid attribute values is incomplete, it may miss invalid values. AI-agent false negatives are less predictable than rule-engine false negatives, because they vary with the model version, the prompt, and the content provided.

Expertise required

Human testing

High. The tester needs working knowledge of WCAG, ARIA, and HTML semantics, plus proficiency with a screen reader.

Rule-engine testing

Low to moderate. Running a tool requires little accessibility knowledge. Interpreting results and determining whether flagged items are genuine violations requires more. Reviewing results across a large ensemble requires the ability to reconcile conflicting findings.

AI-agent testing

Low to moderate for basic use; moderate for reliable use. Formulating precise and effective instructions, understanding the limits of AI output, and critically reviewing findings all benefit from accessibility knowledge.

Best suited for

Human testing

Final verification before release; issues requiring judgment about context, meaning, or usability; interaction flows; screen reader experience; and any issue that requires lived experience with assistive technology.

Rule-engine testing

First-pass scanning across many pages; continuous integration checks; measuring progress over time; identifying well-defined structural violations efficiently; and providing a baseline before human testing.

AI-agent testing

Focused analysis of specific issue types when sufficient page content can be provided; assessment of issues that are too nuanced for mechanical rules but too common for full human review; and generating explanations and remediation guidance.

Introducing the case study

About 5 minutes.

The practicum applies all three strategies to a single type of accessibility issue on a specific web page. The issue type is: incorrect or absent autocomplete attribute on form inputs.

Why the autocomplete attribute matters

The autocomplete attribute tells the browser what kind of personal information an input field collects. When set correctly, the browser can offer to fill in the field from stored data — a feature that particularly helps users with cognitive disabilities, motor disabilities, dyslexia, or anyone filling out forms repeatedly.

WCAG 2.1 and 2.2 Success Criterion 1.3.5 Identify Input Purpose (Level AA) requires that the purpose of inputs collecting personal information can be programmatically determined. The autocomplete attribute is the primary mechanism for satisfying this criterion in HTML.

Valid attribute values

On <input>, <select>, and <textarea> elements, the autocomplete attribute must have a value from the defined list of autofill tokens. Common values include:

• name, given-name, family-name
• email
• tel
• street-address, postal-code, country
• username, new-password, current-password

On a <form> element, the autocomplete attribute is also valid — but with an important restriction: the only valid values for a <form> element are on and off. The richer set of purpose tokens is not valid on <form> elements; those tokens belong on the individual input elements within the form.

Two types of violation

This practicum focuses on two specific violations of WCAG 1.3.5:

• Missing attribute: An input collecting personal information has no autocomplete attribute at all.
• Invalid attribute value: An element has an autocomplete attribute, but its value is not among those recognised by the HTML specification for that element type.

Both violations prevent browsers and assistive technologies from identifying the purpose of the affected fields.

The example page

About 3 minutes.

The example page is the homepage of a nonprofit organisation in the career-guidance sector. It is a publicly accessible, JavaScript-rendered WordPress site.

Relevant page content

The page contains a newsletter sign-up form. This form is not present in the raw HTML that the server delivers. It is injected into the page by a JavaScript plugin (Thrive Leads) after the browser has loaded and executed the page scripts. The form contains one visible user-facing input field:

<input type="email" data-field="email" data-required="1"
  data-validation="email" name="email"
  placeholder="> Enter your email">

The form element wrapping this input is:

<form action="#" method="post" novalidate="novalidate"
  autocomplete="new-password">

Both elements were captured from the rendered DOM on 2026-05-30 using a headless browser. They are the only non-hidden, non-search, non-consent form inputs on the page.

The two defects

Two autocomplete attribute issues are present:

• Issue A — The <input type="email"> element has no autocomplete attribute. Per WCAG 1.3.5, an input that collects an email address requires autocomplete="email".
• Issue B — The <form> element has autocomplete="new-password". On a <form> element, the only valid values are on and off. new-password is a valid token for <input> elements, not for <form> elements. This value appears to be a misapplication of a technique sometimes used to suppress browser autofill on individual input fields.

Human testing

About 7 minutes.

A human tester would approach this page using browser developer tools, keyboard navigation, and (optionally) a screen reader.

What a tester would do

1. Open the page in a browser and let it fully load.
2. Identify interactive elements — in this case, the email input in the newsletter form.
3. Open the browser's developer tools (typically F12 or right-click → Inspect).
4. Select the email input in the Elements panel and examine its attributes.
5. Note that the autocomplete attribute is absent.
6. Select the enclosing <form> element and examine its attributes.
7. Note that autocomplete="new-password" is set on the form.
8. Recall (or look up) that new-password is not a valid value for a <form> element.

Findings

Limitations observed

The form is JavaScript-rendered and appears only after the page fully loads. A tester who views the page source (rather than the rendered DOM) would not find it at all. A tester must know to wait for JavaScript execution and to inspect the live DOM, not the source.

Keyboard navigation and screen reader testing would reveal an additional issue — the email input has no label element (Issue A causes the input to be named only by its placeholder text, which is fragile) — but that is a separate issue not in scope here. Those techniques would not directly reveal the autocomplete attribute issues, which are only visible in the DOM.

Verdict on autocomplete issues

• Issue A (missing attribute on input): Found, if the tester inspects the input element.
• Issue B (invalid value on form): Possibly missed, unless the tester knows the valid values for autocomplete on <form> elements.

Rule-engine testing

About 7 minutes.

Several rule engines were applied to this page. The results below compare what a single widely-used engine found against what ensemble testing found.

Single-engine result: axe-core

axe-core is one of the most widely used open-source accessibility rule engines. It is the engine underlying the axe DevTools browser extension, the popular jest-axe testing library, and many CI/CD integrations. It was run against the fully rendered DOM of the example page in May 2026.

Both autocomplete issues were false negatives for axe-core.

Ensemble result: Kilotest

Kilotest applied 10 tools to the same page in May 2026. Nine of those tools reported at least one issue on the page. Two tools specifically reported autocomplete issues:

• Testaro reported autocomplete missing (Issue A: the email input lacks an autocomplete attribute). WCAG 1.3.5.
• HTML CodeSniffer reported autocomplete invalid (Issue B: the <form> element has autocomplete="new-password", which is not valid for that element type). WCAG 1.3.5.

The full Kilotest report for this page is available at kilotest.com (report 260530T0032/n2u). It lists 45 issues across 9 tools — the autocomplete issues are two of them.

Why the single engine missed both issues

axe-core's autocomplete-valid rule was designed to catch invalid values when an autocomplete attribute is present. It was not designed to enforce WCAG 1.3.5's requirement that inputs collecting personal information must have an autocomplete attribute. For Issue B, axe-core also did not flag the invalid value on the <form> element.

This is not a criticism of axe-core specifically. Every rule engine has coverage gaps. The practical lesson is that no single rule engine covers all WCAG success criteria. Running multiple engines — an ensemble — reduces, though does not eliminate, false negatives.

Verdict on autocomplete issues

• Issue A (missing attribute on input): Missed by axe-core; found by Testaro in the Kilotest ensemble.
• Issue B (invalid value on form): Missed by axe-core; found by HTML CodeSniffer in the Kilotest ensemble.

AI-agent testing

About 7 minutes.

An AI agent was asked to find autocomplete attribute issues on the example page. Two scenarios were tested, illustrating how the content given to the agent affects what it can find.

What the AI agent was given and asked

In both scenarios, the instructions were the same:

Scenario 1: AI agent given the raw HTML source

The raw HTML delivered by the server was provided to the AI agent. This HTML does not contain the newsletter form, which is injected by JavaScript after page load.

Scenario 2: AI agent given the rendered DOM

The HTML of the fully rendered DOM — captured after JavaScript execution — was provided to the AI agent. This HTML includes the newsletter form with its email input and the form element.

What this demonstrates

The two scenarios show that the effectiveness of AI-agent testing depends critically on what the AI agent receives, not only on the capability of the underlying model. Providing the rendered DOM rather than the raw HTML source is the difference between finding both issues and finding neither.

This has a direct practical implication: when using AI agents for accessibility testing, you must understand whether the page's content is server-rendered or JavaScript-rendered. If it is JavaScript-rendered, the agent must receive the rendered DOM, not the raw source.

It also illustrates a limitation that affects all three strategies: all depend on the tester (human, rule engine, or AI agent) receiving the full relevant content. Differences in what each strategy can access — raw source, rendered DOM, accessibility tree, visual rendering — partly determine what each strategy can find.

Verdict on autocomplete issues

• Issue A (missing attribute on input): Missed when given raw HTML; found when given rendered DOM.
• Issue B (invalid value on form): Missed when given raw HTML; found when given rendered DOM.

Comparing the strategies

About 6 minutes.

The practicum produced a clear picture of what each strategy found on the example page:

What each strategy found

Human testing

Found Issue A (missing attribute) on direct inspection. Issue B (invalid value on form) was possibly missed, depending on whether the tester knew that new-password is not a valid value for <form> elements. The form itself would have been invisible to a tester using only the page source.

Rule-engine testing (single engine)

Found neither issue. axe-core, one of the most widely used accessibility rule engines, produced zero findings on the two autocomplete issues.

Rule-engine testing (ensemble)

Found both issues, through two different tools in the ensemble: Testaro found Issue A; HTML CodeSniffer found Issue B.

AI-agent testing (raw HTML)

Found neither issue. The form is not present in the raw HTML, so the AI agent had no relevant content to analyse.

AI-agent testing (rendered DOM)

Found both issues, with accurate explanations and correct remediation advice.

Lessons from the case study