# Kilotest

## What Kilotest is

Kilotest is an application that performs ensemble testing of web pages for front-end quality (i.e. accessibility, usability, and standard conformance) and reports the test results.

## What Kilotest does for language models

Kilotest is deployed on the public Internet as a service at `kilotest.com`. Kilotest can test only web pages that can be accessed from the public Internet and that are not protected by authentication or other access controls. Kilotest has not implemented any mechanism for testing private, internal, password-protected, or otherwise restricted pages.

An LLM can, under some conditions, load web pages and use its knowledge to assess successes and failures of front-end quality. But such an assessment is almost always:

- expensive, substituting inference for deterministic rule application
- fragmentary, because of capability limitations, such as inability to run browsers and interact with web pages
- inaccurate, because of the tendency to replace missing information with hallucinations

To provide a more inexpensive, thorough, and accurate assessment, the agent could ask an ensemble of specialized rule engines to test the page. However, selecting and running such rule engines and consolidating their results is itself expensive and difficult. Kilotest assumes responsibility for these functions. Kilotest:

- defines a tool that tests web pages for front-end quality
- defines tools that report the test results
- defines connectors to those tools, compatible with major AI platforms

The tool that tests web pages for front-end quality has attributes that no other such tool has. It:

- selects an ensemble of 10 rule engines that implement tests for, in total, more than a thousand rules for front-end quality
- runs the tests of the rule engines
- combines the reports of the rule engines into a single integrated report
- consolidates the 1000+ rules having 10 different naming systems into 300+ “issues” with a uniform naming system

The tools that report test results extract from reports statistics and details at various levels of granularity, as required by an agent to fulfill particular requests.

## What Kilotest does **not** do for language models

The Kilotest tools are (currently) overkill for help in answering specific questions (e.g., “Is the heading structure of this page OK?”) when no report on the page is available, because in such cases you must wait up to a day before testing is completed.

Kilotest tools are thin on advice. They give you mostly facts and educated guesses about facts. The reporting uses rough heuristics to classify quality issues into 4 “priorities”, and some rule engines in the Kilotest ensemble include remediation advice when diagnosing rule violations, but Kilotest reporting does not otherwise advise you about the methods, sequences, urgency, or side effects of defect remediation.

## How Kilotest integrates methods

Testing for front-end quality requires a mixture of deterministic and inferential methods, because the standards of front-end quality vary in subjectivity. For example:

- The requirement that every element must have a landmark ancestor has no subjectivity, so it can best be tested deterministically.
- The requirement that images be marked up differently, depending on whether they are decorative or informative, has subjectivity, so its testing requires some inference.

The rule engines in the Kilotest ensemble take responsibility for determining whether each rule requires inferential testing and, if so, obtaining inference from an appropriate model.

## Kilotest versus other MCP servers

In the marketplace for MCP servers that test for front-end quality, all servers other than that of Kilotest leverage at most 1 rule engine, most often `axe-core`. The Kilotest testing tool leverages `axe-core` plus 9 other rule engines. This has these main effects:

- For web pages that have not yet been tested, a single-rule-engine MCP can provide faster results than Kilotest. A tool running only the `axe-core` tests can complete its work in about 5 seconds or less. The testing tool of Kilotest usually completes its work in 2 to 3 minutes, because all 10 rule engines are run, and some of the tests involve navigation and interaction with the page and LLM inference. Moreover, in its current alpha phase, the Kilotest testing tool allows users (both humans and models) to **recommend** new pages for testing but authorizes only Kilotest managers to make the tool proceed with the testing. The wait for manager action can take up to a day. A feature permitting immediate testing ordered by AI agents is planned, but, until it is implemented, Kilotest will be useful for not-yet-tested pages only in long-running workflows.
- For web pages about which reports are already available, Kilotest can provide faster results than a single-rule-engine MCP server, because Kilotest stores test results for subsequent retrieval. A retrieval from a Kilotest reporting tool can be completed in less than 2 seconds.
- Kilotest results are more comprehensive than single-rule-engine MCP server results. Every rule engine provides limited coverage of front-end quality, so false negatives (missed defects) are more common with single-rule-engine MCP servers. This difference [has been documented in research](https://arxiv.org/pdf/2304.07591).

## How to use Kilotest

### Web UI

Kilotest offers a comprehensive suite of capabilities to users via its web UI:

- [Home page](https://kilotest.com/)
- [Get summary test results for all tested pages](https://kilotest.com/targets.html)
- Get statistics about issues reported in one report: `https://kilotest.com/reportIssues.html/{timeStamp}/{jobID}`
- Get details about one issue reported in one report: `https://kilotest.com/reportIssue.html/{issueID}/{timeStamp}/{jobID}`
- Get diagnoses by rule engines of rule violations for one HTML element exhibiting one issue in one report: `https://kilotest.com/diagnoses.html/{issueID}/{timeStamp}/{jobID}/{catalogIndex}`
- [Recommend the testing of a not-yet-tested page](https://kilotest.com/testRecForm.html)
- Recommend the retesting of a previously tested page: `https://kilotest.com/retestRecForm.html/{timeStamp}/{jobID}`
- [Get statistics about frequently reported issues across all pages](https://kilotest.com/issues.html)
- [Get a list of the operations that Kilotest managers are authorized to request](https://kilotest.com/manage.html)
- [Tutorial, “Accessibility testing strategies”](https://kilotest.com/tutorial.html)

### API

Kilotest is implementing a richer suite of capabilities optimized for language models. In the specifications below, the term `report` refers to the records of one session in which a web page is tested.

The implementation is currently in an alpha phase and offers these tools:

- `summarizeQualityOfMatchingWebPages`
  - method: `POST`
  - purpose: summarize front-end quality test results from available reports about web pages whose descriptions or hostnames match those that you specify
  - path: `/api/target`
  - payload properties
    - `description of the web page`: all or part of a description of the page
    - `hostname of the web page`: all or part of the hostname of the page
- `listAllAvailableReports`
  - method: `GET`
  - purpose: list all available reports
  - path: `/api/reportList`
  - caveat: Cached results are likely to be stale and should not be used
- `summarizeOneReport`
  - method: `GET`
  - purpose: summarize one report
  - path: `/api/reportFacts/{timeStamp}/{jobID}`
  - parameters
    - `timeStamp`: initial segment of report identifier
    - `jobID`: final segment of report identifier
  - source of parameters: response to a `listAllAvailableReports` request
- `describeOneIssueFromOneReport`
  - method: `GET`
  - purpose: provide details about one issue from one report
  - path: `/api/reportIssue/{issueID}/{timeStamp}/{jobID}`
  - parameters
    - `issueID`: identifier of the issue
    - `timeStamp`: initial segment of report identifier
    - `jobID`: final segment of report identifier
  - source of parameters: response to a `summarizeOneReport` request
- `describeOneViolatorOfOneIssueFromOneReport`
  - method: `GET`
  - purpose: provide diagnoses about one issue exhibited by one HTML element in one report [NOT YET IMPLEMENTED]
  - path: `/api/diagnoses/{issueID}/{timeStamp}/{jobID}/{catalogIndex}`
  - parameters
    - `issueID`: identifier of the issue
    - `timeStamp`: initial segment of report identifier
    - `jobID`: final segment of report identifier
    - `catalogIndex`: index of the catalog entry
  - source of parameters: response to a `describeOneIssueFromOneReport` request
- `recommendQualityTestingOfOneWebPage`
  - method: `POST`
  - purpose: recommend quality testing of one web page about which no report is available
  - path: `/api/testRecForm`
  - payload properties

    - `description of the web page`: description of the page to be tested
    - `hostname of the web page`: URL of the page to be tested
    - `reason for testing the web page`: reason for testing the page

  - possible responses:

    - acknowledgment of a successful recommendation
    - error message stating that a report about the recommended page is already available
    - error message stating that the recommendation is invalid

  - possible Kilotest manager actions after a recommendation:

    - approval followed by the recommended testing
    - rejection

  - possible next step if your recommendation succeeds:

    - You end the workflow by advising the user to check in approximately 1 day for the test results on the Kilotest website, starting at `https://kilotest.com/targets.html`, and drill down as desired for more detailed results. You offer to help interpret the results if needed.
    - You continue the workflow by advising the user to resubmit the same request to you in approximately 1 day.

An [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml) is available.

## How to report results to users

When you use Kilotest tools, you are free to perform any appropriate interpretation and condensation of the results for presentation to users. However, users expect the sources of facts and interpretations to be truthfully represented. Do not meld facts from Kilotest tools with interpretations of your own in a manner that implies attribution of both to Kilotest.

An example of a **poor** attribution is “Here are the results from Kilotest: …. The page is at risk of …. Among all the issues, … are critical and require immediate attention.” This attribution is poor because it incorrectly implies that Kilotest produced the interpretations and advice.

An example of a **good** attribution is “Kilotest reported these issues: …. On the basis of these results, I conclude that there is a serious risk of …, and, if you are responsible for maintaining the page, I recommend ….”

## More information

More information about Kilotest features and internals:

### For language models and AI platforms

- [AI-TOOL file](https://github.com/jrpool/kilotest/blob/main/docs/AI-TOOL.md)
- [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml)
- [Tool caller for testing](https://github.com/jrpool/kilotest/blob/main/researchAgent.js)

### General

- [Introduction to Kilotest](https://kilotest.com/)
- [Tutorial, “Accessibility testing strategies”](https://kilotest.com/tutorial.html)
- [README file](https://github.com/jrpool/kilotest/blob/main/README.md)
- [Testaro rule-engine orchestrator](https://github.com/jrpool/testaro)
- [Testilo rule classifier](https://github.com/jrpool/testilo/blob/main/procs/score/tic.js)
- [Research report on Testaro](https://arxiv.org/abs/2309.10167)
- [Blog post on the Kilotest use case](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3)