WCAG21 validation in build

Last modified by Vincent Massol on 2024/02/26 17:54



WCAG 2.1

First of all, it's important to note that WCAG 2.1 specifications only extend WCAG 2.0 (and WCAG 1.0). Those versions are incremental.

There are three levels of compliance for WCAG:

  • A: Minimal requirements, for the most part easy to test automatically
  • AA: Standard requirement, for the most part easy to test automatically
  • AAA: strict requirements, rarely needed and for the most part tests needs to be performed by a subject.

The WCAG 2.1 AA specifications has proved itself to be a valued standard for accessibility.

The refle

WCAG 2.1 Validation Tool Choice

Tool listing

When filtering through the Web Accessibility Evaluation List, and restricting the tools to those which support WCAG 2.1 guidelines and have an open source license, 16 results are found:


The main components we want for a tool to integrate WCAG 2.1 AA rules in the XWiki Webstandard validation tests are:

  • Automatable in a build (needed)
  • Configurability
  • Activity
    • Commit frequency
    • Amount of contributors
  • Release frequency
  • Extensibility

Wide scope comparison

IDToolAutomatable (Y/N)ConfigurabilityActivity - Commit frequencyActivity - Amount of contributorsRelease frequencyExtensibilityNotes


A11y sitecheckerY++--- none since February 2022-- 6 total-- irregular, from 1 every 2 days to 1 year without any+Depends on axe-core/puppeteer; only one npm package dependant.
2A11ygatoY++------NAN---Archived project in November 2022. Fork of pally (dependant of axe-core and codesniffer).
3A11yWatchY+++++ 20 commits in last month--- 3 total, 1 active- irregular, from 1 every 2 days to 6 months without any++Comes with clients in a lot of languages.
4Accessibility Insights for WebN+++++50 commits in last month++ 53 total, 13 in last month+++ ~1 month+Browser plugin (Chrome and MS Edge) and Android app checker. Based on axe-core.
5Accessibility SuiteN----- 2 years without any update--- 1 totalNANNANWordpress Plugin. The W3 link redirects to the current commercial page. Seems like most of the content is as of now locked behind a commercial licence: the plugin reviews.
6ANDIN--- ~5 commits per month-- 2 total, 1 activeNAN---Bookmarklet for most common browsers.
7axe-core - accessibility testing rules libraryY+++++ 261commits/y+ 190 total, 6 in last month+++ ~2 months++Does not return false positive. Main contributor is also one of the core contributors of ACT Rules. Used by 4.5m.
8Domain Accessibility AuditY+++--- none since October 2021- 4 totalNAN-Based on axe-core.
9IBM Equal Access Accessibility Checker (Chrome and Firefox extension)N++++ 30 commits in last month++ 25 total, 8 last month+++ ~1 month+Activity measured on https://github.com/IBMa/equal-access . Used by 250.
Browser plugin (Chrome & Firefox).
10IBM Equal Access Karma Accessibility CheckerY++++++ 30 commits in last month++ 25 total, 8 last month+++ ~1 month+Activity measured on https://github.com/IBMa/equal-access . Used by 250.
11IBM Equal Access NPM Accessibility CheckerY++++++ 30 commits in last month++ 25 total, 8 last month+++ ~1 month+Activity measured on https://github.com/IBMa/equal-access . Used by 250.
12NerdeFocusNNAN--- none since January 20222 totalNANNANBrowser plugin (Chrome). Scope limited to focus, and no validation of WCAG.
13NerdeRegionNNAN--- none since May 20221 totalNANNANBrowser plugin (Chrome). Scope limited to aria regions, and no validation of WCAG.
14QualWebY+++-- 20 commits last year- 8 total, 3 activeNAN++Implements WCAG 2.1 and ACT Rules.
15UDOITN-+ 6 commits in last month+ 18 total, 2 active++ ~6 months--Scan of Canvas courses.
16WhoCanUseN-+ 40 commits last year, none since October 2022+ 11 total, 2 activeNAN-Scope limited to contrast, and no validation of WCAG.

We can see in this table that out of the 16 solutions, the first important differentiation factor is the ability of the tools for automated testing. Only A11y sitechecker (1), A11ygator (2), A11yWatch (3), axe-core (7), Domain accessibility audit

(8), IBM equal access Karma (10) & NPM (11) accessibility checkers and Qualweb (14) fulfill our basic needs.

Close scope comparison

For each tool and each researched property, a grade ranging from +++ to --- (or NAN) evaluates its overall level. Those grades are summed up in a result column, with a lower ponderation for the "Activity" sub-properties.

IDToolConfigurabilityActivity - Commit frequencyActivity - Amount of contributorsRelease frequencyExtensibilityGradeNotes


A11y sitechecker++--- none since February 2022-- 6 total-- irregular, from 1 every 2 days to 1 year without any+-3Depends on axe-core/puppeteer; only one npm package dependant.
2A11ygato++------NAN----14Archived project in November 2022. Fork of pa11y (dependant of axe-core and codesniffer).
3A11yWatch+++++ 20 commits in last month--- 3 total, 1 active- irregular, from 1 every 2 days to 6 months without any+++7Comes with clients in a lot of languages.
7axe-core - accessibility testing rules library+++++ 20commits/month++ 190 total, 6 in last month+++ ~2 months+++19Does not return false positive. Main contributor is also one of the core contributors of ACT Rules. Used by 4.5m.
8Domain Accessibility Audit+++--- none since October 2021- 4 totalNAN--6Based on axe-core.
10IBM Accessibility Checkers++++++ 30 commits in last month++ 25 total, 8 last month+++ ~1 month++19Activity measured on https://github.com/IBMa/equal-access . Used by 250.
14QualWeb+++-- 20 commits last year- 8 total, 3 activeNAN+++1

Implements WCAG 2.1 and ACT Rules.

However after some research, it seems like pa11y , from which is forked A11ygato (2), is also a valid automatic testing tool, and at that, much more active than A11ygato itself. Pa11y was not found in the initial research on W3 because it was only tagged as WCAG 2.0 (and not 2.1). That's because the tool kept all of its documentation as "WCAG2" without former precision. However as of January 2023, according to the source code, they had switched to WCAG 2.1.

IDToolConfigurabilityActivity - Commit frequencyActivity - Amount of contributorsRelease frequencyExtensibilityGradeNotes


Pa11y+++- 10 commits last year, none since October 2022++ 60 total, 3 active-- irregular, none since April 2022++5Depends on axe-core and codesniffer. Used by 5.5k .

From all these results we can filter out a couple of frameworks to use for WCAG 2.1 validation in XWiki:


After considering the scope and possibilities of both the tools, axe-core has been kept:

  • More issues found than IBM Accessibility Checkers on the XWiki Main page.
  • Issues are given not only a status (violation, incomplete, ...) but also a scope (critical, serious, major, minor).
  • Best practices to follow are also reported.
  • Used in most of the accessibility validation web extensions

Proposal for a WCAG2.1 Validation framework: axe-core .

WCAG 2.1 Validation Integration


The integration of validation with axe-core does not need a new testing framework. Moreover, coverage of interactive elements needs to be done using the previous tests written. That's why we can complete the current testing framework (based on testcontainer, docker and selenium) with the accessibility assesment.

Accessibility validation is quite slow, that's why we deactivate it by default on builds, and only activate it when a parameter is set to true: xwiki.test.ui.wcag .

This parameter will trigger some setup and an assert for the whole test suite in XWikiDockerExtension .

The accessibility test itself will be done during the instantiation of any Base Element. Base elements are the core of the current testing sets and making sure we validate the content of every new baseElement is enough to cover all pages (since all pages should be accompagnied by their own functionnal test set) and all interactions.


  • Add dependencies in the pom.xml files for the xwiki-platform-test-ui and xwiki-platform-test-docker packages:<dependency>

    Update the version of the dependency to io.github.bonigarcia if needed.

  • At each BasePage instanciation, if the wcag parameter is on, analyze its web driver using axe and write the result in a cache in the permanentContext.
    Note: at first, we thought validating in the Base Element constructor was a good idea, however, as pointed out here , there is an issue with elements who need some time to be loaded.
    That's why we decided to move the validation at the end of the BasePage constructor. This way, even though there is less checks, they will be conducted properly.
  • In TestUtils, create and manage a WCAGContext that holds all the info we want to gather about accessibility validation on this test suite.
  • In XWikiDockerExtension, At the end of the test suite, if the validation is ON, make an assert on the amount of violations (accumulated during tests in TestUtils) and report different infos on the validation.
    • Before any test, transfers the xwiki.test.ui.wcag system parameter to TestUtils so it can be read during the tests by the Base Pages.
    • Reports from all the validations are merged afterAll the tests in the test suite, writes reports to proper files and asserts that no failing violation is found.
  • In UITestTestConfigurationResolver, UITest and TestConfiguration, set up the xwiki.test.ui.wcag system parameter to use in XWikiDockerExtension.

WCAGContext Class

The WCAGContext class contains all information about the wcag reports in a test suite. It can be a part of org.xwiki.test.ui.

  • Once a validation is made, a call to addWcagResults is made. This function is central to the class report.
    • It first verifies that validation has been done on a non already cached element.
    • Then it instantiates a new WCAGTestResults from the current context. This subclass is in charge of creating and giving access to the two reports generated from a validation.
      Reports are stored based on the level: warning vs error.
      Reports are stored as Strings.
    • At last, it updates the cache to make sure validation won't happen twice on the same object.
  • Timing can be kept for each validation and validation time can be summed up at the end of the test suite for performance auditing.
  • A cache can be kept to avoid testing multiple times the same page. The cost of a WCAG validation is heavy, and some test sets are repetitive, so this can save a lot of time.
    The cache checks if there's a match in already tested couples [URL:pageClass].
    Note: Some states of the pages reached only after manipulations won't be validated. This is a drawback to this cache system.

A few static final lists are set in the context. Those lists are not meant to be changed on every other build, however they should change to account for changes to WCAG and in our management of accessibility violations:

  • FAILS_ON_RULE : short term changes, sets the level of the rule violation. See here for more details.
  • VALIDATE_TAGS : long term changes, related to the wcag context. Update this when new axe-core tags corresponding to up-to-date WCAG standards are available.
  • DISABLED_RULES : long term changes, related to what wcag rules are not checked. Should be empty, unless wcag validation for some rules is done elsewhere.
    Note: Might be used if a wcag specific test is set up separately, e.g. to validate color related rules with a proper colortheme.

Separating fails and warnings

We want to ensure a smooth integration of the wcag validation in the build.

In order to do so, all rules in our test set are checked, but at the start of the process none will fail the build.

One rule ID after the other, once we make sure there is no warning in the build for this ruleID, we can set its WCAGContext.FAILS_ON_RULE entry to true.

A WCAGContext.FAILS_ON_RULE entry to false is functionally useless, however it reports that issues concerning this rule need to be solved before making it a failing rule.

By tagging a rule as failing, from this point on, we can fail the build for this rule if there's any regression.

This system ensures a rule by rule control over accessibility fails in the build.

When a new rule is added to the validation set of rules (by the use of a new axe-core tag e.g. "wcag22aa"), this is the process to handle it:

  • (By default, the rule will be non failing, during builds, any violation of this rule will be reported in the `wcagWarnings` file.)
  • Fix every violation of this rule in the build.
  • Create a new entry in the WCAGContext.FAILS_ON_RULE set to true in order to make this rule failing.

From this point on, any regression on the new rule will fail the build.


A proof of concept implementation has been done using the axe-core-maven-html library.

We picked the use case of 10 \@UITests. They all trigger the instantiation of at least one BasePage:

  1. DashboardIT
    Test suite total: 1 a11y check
    1. editDashboard
      line 51: gotoPage
      line 53: new DashboardEditPage , NO ally check since there is no inheritance from BasePage
  2. MenuIT
    Test suite total: 16 a11y checks
    1. verifyMenuInApplicationsIndex
      line 66: gotoPage
      line 69: applicationIndexHomePage.clickApplication
      line 78: gotoPage
    2. verifyMenuCreationInLeftPanelWithCurrentWikiVisibility
      line 93: gotoPage
      line 98: pane.clickAdd
      line 99: new MenuEntryEditPage()
      line104: meep.clickSaveAndView()
      line 111: gotoPage
    3. verifyMenuIsAvailableInAdministration
      line 135: gotoPage
      line 140: administrationPage.clickSection
      line 143: new MenuHomePage
      line 152 pane.clickAdd
      line 157: new MenuEntryEditPage
      line 162: meep.clickSaveAndView
      line 169: gotoPage
  3. LiveDataIT
    Test suite  total: 8 a11y checks               
    1. livedataLivetableTableLayout
      line 136:  new ClassEditPage
      line 136: classEditPage.clickSaveAndView
      line 140: testUtils.createPage
      line 150: testUtils.createPage
                       line 153: gotoPage
      line 215: gotoPage
    2.     livedataLivetableTableLayoutResultPage
      line 264: initResultPage
      line 270: gotoPage
  4. Administr ationIT
    Test suite total: 6 a11y checks
    1. verifyAdministrationSections
      line 51: gotoPage
      line 57: new AdministrablePage
      line 58: page.clickAdministerWiki
          line 74: gotoPage
      line 75: new AdminisitrablePage
      line 76: page.clickAdministerPage
  5. RegisterIT
    Test suite total: 28 a11y checks
    These are parameterizedTests, which take two parameters. The first one is transparent concerning accessibility validation, the second one changes some of the loaded PageElements.

    Each test will play three times, and both cases (true/false) are covered in these situations. Since the cache is in place, we don't mind at all the third repetition, no a11y validaiton will run in it.
    These tests might be migrated to some page tests at one point, since they for the most part check minimal changes in the values in the form.

    1. RegisterJohnSmith
      line 87 (setup):
      line 175 (this.getRegistrationPage(isModal) )
      isModal = true:
      line 205 (RegistrationModal.gotoPage)
      line 23: UsersAdministrationSectionPage.gotoPage -> 2 checks
      line 25: clickAddNewUser
      isModal = false:
      line 207 (RegistrationPage.gotoPage)
      line 42: new RegistrationPage
    2. registerExistingUser
      SAME (4 checks)
    3. registerPasswordTooShort
      SAME (4 checks)
    4. registerDifferentPasswords
      SAME (4 checks)
    5. registerEmptyPassword
      SAME (4 checks)
    6. registerEmptyUserName
      SAME (4 checks)
    7. registerInvalidEmail
      SAME (4 checks)
  6. TipsPanelIT
    Test suite total: 4 a11y checks
    1. VerifyTipsParameterIsRestricted
      line 80: gotoPage
      line 81: new PanelViewPage
    2. verifyTipsContentIsExecutedWithTheRightAuthor
      line 114: gotoPage
      line 115: new PanelViewPage
  7. AttachmentGalleryPickerMacroIT
    Test suite total: 2 a11y checks
    1. AttachmentGalleryPickerMacro
      line 71: setup.createPage
      line 76: new AttachmentsViewPage
  8. CommentsIT
    Test suite total: 4 a11y checks
    1. CommentAsGuest
      line 69: setup.createPage
    2. commentAsLoggedInUser
      line 91: setup.createPage
    3. commentAsAdmin
      line 145: setup.createPage
      line 157: setup.gotoPage
  9. UserProfileIT
    Test suite total: 65 a11y checks
    @BeforeEach: 1 check
    line 117: new EditPage
    1. editProfile
      line 125: ProfileUserProfilePage.gotoPage -> 2 checks
      line 126: userProfilePage.editProfile
      line 136: profileEditPage.clickSaveAndView
      line 138: new ProfileUserProfilePage
    2. ChangeAvatarImage
      line 156: ProfileUserProfilePage.gotoPage -> 2 checks
      line 157: userProfilePage.changeAvatarImage
    3. ChangeUserProfile
      line 169: ProfileUserProfilePage.gotoPage -> 2 checks
      line 170: userProfilePage.switchToPreferences
      line 185: preferencesPage.editPreferences
      line 187: preferencesEditPage.clickSaveAndView
      line 189: new ProfileUserProfilePage
      line 190: userProfilePage.switchToPreferences
    4. ChangeDefaultEditor
      line 199: ProfileUserProfilePage.gotoPage -> 2 checks
      line 200: userProfilePage.switchToPreferences
      line 203: preferencesPage.editPreferences
      line 205: preferencesEditPage.clickSaveAndView
      line 207: new ProfileUserProfilePage
      line 208: userProfilePage.switchToPreferences
      line 212: ProfileUserProfilePage.gotoPage -> 2 checks
      line 213: userProfilePage.switchToPreferences
      line 214: preferencesPage.editPreferences
      line 216: preferencesEditPage.clickSaveAndView
      line 218: new ProfileUserProfilePage
      line 219: userProfilePage.switchToPreferences
      line 223: ProfileUserProfilePage.gotoPage -> 2 checks
      line 224: userProfilePage.switchToPreferences
      line 225: preferencesPage.editPreferences
      line 227: preferencesEditPage.clickSaveAndView
      line 229: new ProfileUserProfilePage
      line 230: userProfilePage.switchToPreferences
    5. CommentDoesntOverrideAboutInformation
      line 242: ProfileUserProfilePage.gotoPage -> 2 checks
    6. EnsureDashboardUIAddAnObjectAtFirstEdit
      line 259: ProfileUserProfilePage.gotoPage -> 2 checks
    7. VerifyGroupTab
      line 269: GroupsUserProfilePage.gotoPage
    8. ToggleEnableDisable
      line 283: ProfileUserProfilePage.gotoPage -> 2 checks
      line 290: ProfileUserProfilePage.gotoPage -> 2 checks
      line 300: ProfileUserProfilePage.gotoPage -> 2 checks
      line 309: ProfileUserProfilePage.gotoPage -> 2 checks
    9. disabledUserTest
      line 320: ProfileUserProfilePage.gotoPage -> 2 checks
      line 332: setup.gotoPage
  10. NotificationsIT
    Test suite total: 84 a11y checks
    @BeforeAll: 4 checks
    line 102: NotificationsUserProfilePage.gotoPage
    line 106: new NotificationsUserProfilePage
    line 112: NotificationsUserProfilePage.gotoPage
    line 116: new NotificationsUserProfilePage
    1. simpleNotifications
      line 137: setup.createPage
      line 140: setup.gotoPage
      line 142: new NotificationsTrayPage
      line 146: NotificationsUserProfilePage.gotoPage
      line 154: setup.createPage -> loop * 19
      line 156: setup.createPage
      line 160: setup.gotoPage
      line 163: new NotificationsTrayPage
      line 175: setup.gotoPage
      line 178: new NotificationsTrayPage
      line 190: new NotificationsTrayPage
      line 195: NotificationsUserProfilePage.gotoPage
      line 205: setup.gotoPage
      line 208: new NotificationsTrayPage
    2. compositeNotifications
      line 221: NotificationsUserProfilePage.gotoPage
      line 229: setup.gotoPage
      line 230: new NotificationsTrayPage
      line 235: setup.createPage
      line 237: setup.gotoPage
      line 239: new WikiEditPage
      line 242: new ViewPage
      line 244: new WikiEditPage
      line 247: setup.gotoPage
      line 253: setup.gotoPage
      line 255: new NotificationsTrayPage
    3. notificationDisplayerClass
      line 293: setup.gotoPage
      line 294: setup.createPage
      line 297: setup.createPage
      line 307: setup.editObjects
      line 319: NotificationsUserProfilePage.gotoPage
      line 327: setup.gotoPage
      line 329: new WikiEditPage
      line 335: setup.gotoPage
      line 339: new NotificationsTrayPage
    4. ownEventNotifications
      line 357: NotificationsUserProfilePage.gotoPage
      line 364: setup.createPage
      line 367: new NotificationsTrayPage
      line 374: NotificationsUserProfilePage.gotoPage
      line 379: setup.createPage
      line 380: setup.gotoPage
      line 381: new NotificationsTrayPage
    5. guestUsersDontSeeNotificationMenu
      line 402: setup.gotoPage
      line 403: new NotificationsTrayPage
      line 405: new NotificationsTrayPage

All the totals are calculated without taking the cache into account. Some of these checks will be short-wired by an already existing cache entry.

The first thing to consider is the relatively slow WCAG validation time. For each base page such a validation takes between 2 and 6 seconds (axe-core has a timeout of 30 seconds that can be reached on very large pages).

This validation time makes the integration tests significantly slower: on this relatively small test suite, it added up to almost two minutes.

However, there must be solutions to this issue https://www.deque.com/axe/core-documentation/api-documentation/#section-4-performance . Moreover, it's possible to restrict the content analyzed for each base element, making it faster overall.

The reports from axe contain plenty of information, that can be filtered and organized to fulfill our needs (https://github.com/dequelabs/axe-core-maven-html/blob/develop/utilities/src/main/java/com/deque/html/axecore/results/AxeResults.java). In this PoC, only the list of Rules violated are kept and concatenated https://github.com/dequelabs/axe-core-maven-html/blob/develop/utilities/src/main/java/com/deque/html/axecore/results/Rule.java .

Testing it out

The source code for this PoC is available here: https://github.com/Sereza7/xwiki-platform/tree/XWIKI-20541

xwiki-platform-test-docker and xwiki-platform-test-ui should be built.

In order to test it out, you can use this command in the xwiki-platform-menu-test-docker package:

mvn -Dxwiki.test.ui.wcag=true -Dgradle.cache.local.enabled=false -Dgradle.cache.remote.enabled=false -Dtest=MenuIT clean verify

Removing the first parameter will default to no accessibility validation, this can give you an idea of the duration of the test suite.


Get Connected