Description

• WCAG 2.1
• WCAG 2.1 Validation Tool Choice
  • Tool listing
  • Requirements
  • Wide scope comparison
  • Close scope comparison
  • Conclusion
• WCAG 2.1 Validation Integration
  • Abstract
  • Description
    • WCAGContext Class
    • Separating fails and warnings
  • Result
  • Testing it out

WCAG 2.1

First of all, it's important to note that WCAG 2.1 specifications only extend WCAG 2.0 (and WCAG 1.0). Those versions are incremental.

There are three levels of compliance for WCAG:

• A: Minimal requirements, for the most part easy to test automatically
• AA: Standard requirement, for the most part easy to test automatically
• AAA: strict requirements, rarely needed and for the most part tests needs to be performed by a subject.

The WCAG 2.1 AA specifications has proved itself to be a valued standard for accessibility.

The refle

WCAG 2.1 Validation Tool Choice

Tool listing

When filtering through the Web Accessibility Evaluation List, and restricting the tools to those which support WCAG 2.1 guidelines and have an open source license, 16 results are found:

• A11y sitechecker
• A11ygato
• A11yWatch
• Accessibility Insights for Web
• Accessibility Suite
• ANDI
• axe-core - accessibility testing rules library
• Domain Accessibility Audit
• IBM Equal Access Accessibility Checker (Chrome and Firefox extension)
• IBM Equal Access Karma Accessibility Checker
• IBM Equal Access NPM Accessibility Checker
• NerdeFocus
• NerdeRegion
• QualWeb
• UDOIT
• WhoCanUse

Requirements

The main components we want for a tool to integrate WCAG 2.1 AA rules in the XWiki Webstandard validation tests are:

• Automatable in a build (needed)
• Configurability
• Activity
  • Commit frequency
  • Amount of contributors
• Release frequency
• Extensibility

Wide scope comparison

We can see in this table that out of the 16 solutions, the first important differentiation factor is the ability of the tools for automated testing. Only A11y sitechecker (1), A11ygator (2), A11yWatch (3), axe-core (7), Domain accessibility audit

(8), IBM equal access Karma (10) & NPM (11) accessibility checkers and Qualweb (14) fulfill our basic needs.

Close scope comparison

For each tool and each researched property, a grade ranging from +++ to --- (or NAN) evaluates its overall level. Those grades are summed up in a result column, with a lower ponderation for the "Activity" sub-properties.

However after some research, it seems like pa11y , from which is forked A11ygato (2), is also a valid automatic testing tool, and at that, much more active than A11ygato itself. Pa11y was not found in the initial research on W3 because it was only tagged as WCAG 2.0 (and not 2.1). That's because the tool kept all of its documentation as "WCAG2" without former precision. However as of January 2023, according to the source code, they had switched to WCAG 2.1.

From all these results we can filter out a couple of frameworks to use for WCAG 2.1 validation in XWiki:

• Appropriate candidates:
  • axe-core - accessibility testing rules library (+19)
  • IBM Accessibility Checkers (+19)
• Plausible candidates
  • A11yWatch (+7)
  • Pa11y (+5)
• Possible candidates
  • QualWeb (+1)
  • A11y sitechecker (-3)
  • Domain Accessibility Audit (-6)

Conclusion

After considering the scope and possibilities of both the tools, axe-core has been kept:

• More issues found than IBM Accessibility Checkers on the XWiki Main page.
• Issues are given not only a status (violation, incomplete, ...) but also a scope (critical, serious, major, minor).
• Best practices to follow are also reported.
• Used in most of the accessibility validation web extensions

Proposal for a WCAG2.1 Validation framework: axe-core .

WCAG 2.1 Validation Integration

Abstract

The integration of validation with axe-core does not need a new testing framework. Moreover, coverage of interactive elements needs to be done using the previous tests written. That's why we can complete the current testing framework (based on testcontainer, docker and selenium) with the accessibility assesment.

Accessibility validation is quite slow, that's why we deactivate it by default on builds, and only activate it when a parameter is set to true: xwiki.test.ui.wcag .

This parameter will trigger some setup and an assert for the whole test suite in XWikiDockerExtension .

The accessibility test itself will be done during the instantiation of any Base Element. Base elements are the core of the current testing sets and making sure we validate the content of every new baseElement is enough to cover all pages (since all pages should be accompagnied by their own functionnal test set) and all interactions.

Description

• Add dependencies in the pom.xml files for the xwiki-platform-test-ui and xwiki-platform-test-docker packages:<dependency>
   <groupId>com.deque.html.axe-core</groupId>
   <artifactId>selenium</artifactId>
   <version>4.6.0</version>
  </dependency>

  Update the version of the dependency to io.github.bonigarcia if needed.

• At each BasePage instanciation, if the wcag parameter is on, analyze its web driver using axe and write the result in a cache in the permanentContext.
  Note: at first, we thought validating in the Base Element constructor was a good idea, however, as pointed out here , there is an issue with elements who need some time to be loaded.
  That's why we decided to move the validation at the end of the BasePage constructor. This way, even though there is less checks, they will be conducted properly.
• In TestUtils, create and manage a WCAGContext that holds all the info we want to gather about accessibility validation on this test suite.
• In XWikiDockerExtension, At the end of the test suite, if the validation is ON, make an assert on the amount of violations (accumulated during tests in TestUtils) and report different infos on the validation.
  • Before any test, transfers the xwiki.test.ui.wcag system parameter to TestUtils so it can be read during the tests by the Base Pages.
  • Reports from all the validations are merged afterAll the tests in the test suite, writes reports to proper files and asserts that no failing violation is found.
• In UITestTestConfigurationResolver, UITest and TestConfiguration, set up the xwiki.test.ui.wcag system parameter to use in XWikiDockerExtension.

WCAGContext Class

The WCAGContext class contains all information about the wcag reports in a test suite. It can be a part of org.xwiki.test.ui.

• Once a validation is made, a call to addWcagResults is made. This function is central to the class report.
  • It first verifies that validation has been done on a non already cached element.
  • Then it instantiates a new WCAGTestResults from the current context. This subclass is in charge of creating and giving access to the two reports generated from a validation.
    Reports are stored based on the level: warning vs error.
    Reports are stored as Strings.
  • At last, it updates the cache to make sure validation won't happen twice on the same object.
• Timing can be kept for each validation and validation time can be summed up at the end of the test suite for performance auditing.
• A cache can be kept to avoid testing multiple times the same page. The cost of a WCAG validation is heavy, and some test sets are repetitive, so this can save a lot of time.
  The cache checks if there's a match in already tested couples [URL:pageClass].
  Note: Some states of the pages reached only after manipulations won't be validated. This is a drawback to this cache system.

A few static final lists are set in the context. Those lists are not meant to be changed on every other build, however they should change to account for changes to WCAG and in our management of accessibility violations:

• FAILS_ON_RULE : short term changes, sets the level of the rule violation. See here for more details.
• VALIDATE_TAGS : long term changes, related to the wcag context. Update this when new axe-core tags corresponding to up-to-date WCAG standards are available.
• DISABLED_RULES : long term changes, related to what wcag rules are not checked. Should be empty, unless wcag validation for some rules is done elsewhere.
  Note: Might be used if a wcag specific test is set up separately, e.g. to validate color related rules with a proper colortheme.

Separating fails and warnings

We want to ensure a smooth integration of the wcag validation in the build.

In order to do so, all rules in our test set are checked, but at the start of the process none will fail the build.

One rule ID after the other, once we make sure there is no warning in the build for this ruleID, we can set its WCAGContext.FAILS_ON_RULE entry to true.

A WCAGContext.FAILS_ON_RULE entry to false is functionally useless, however it reports that issues concerning this rule need to be solved before making it a failing rule.

By tagging a rule as failing, from this point on, we can fail the build for this rule if there's any regression.

This system ensures a rule by rule control over accessibility fails in the build.

When a new rule is added to the validation set of rules (by the use of a new axe-core tag e.g. "wcag22aa"), this is the process to handle it:

• (By default, the rule will be non failing, during builds, any violation of this rule will be reported in the `wcagWarnings` file.)
• Fix every violation of this rule in the build.
• Create a new entry in the WCAGContext.FAILS_ON_RULE set to true in order to make this rule failing.

From this point on, any regression on the new rule will fail the build.

Result

A proof of concept implementation has been done using the axe-core-maven-html library.

We picked the use case of 10 \@UITests. They all trigger the instantiation of at least one BasePage:

1. DashboardIT
   Test suite total: 1 a11y check
   1. editDashboard
      line 51: gotoPage
      line 53: new DashboardEditPage , NO ally check since there is no inheritance from BasePage
2. MenuIT
   Test suite total: 16 a11y checks
   1. verifyMenuInApplicationsIndex
      line 66: gotoPage
      line 69: applicationIndexHomePage.clickApplication
      line 78: gotoPage
   2. verifyMenuCreationInLeftPanelWithCurrentWikiVisibility
      line 93: gotoPage
      line 98: pane.clickAdd
      line 99: new MenuEntryEditPage()
      line104: meep.clickSaveAndView()
      line 111: gotoPage
       
   3. verifyMenuIsAvailableInAdministration
      line 135: gotoPage
      line 140: administrationPage.clickSection
      line 143: new MenuHomePage
      line 152 pane.clickAdd
      line 157: new MenuEntryEditPage
      line 162: meep.clickSaveAndView
      line 169: gotoPage
3. LiveDataIT
   Test suite  total: 8 a11y checks               
   1. livedataLivetableTableLayout
      line 136:  new ClassEditPage
      line 136: classEditPage.clickSaveAndView
      line 140: testUtils.createPage
      line 150: testUtils.createPage
                       line 153: gotoPage
      line 215: gotoPage
   2.     livedataLivetableTableLayoutResultPage
      line 264: initResultPage
      line 270: gotoPage
4. Administr ationIT
   Test suite total: 6 a11y checks
   1. verifyAdministrationSections
      line 51: gotoPage
      line 57: new AdministrablePage
      line 58: page.clickAdministerWiki
          line 74: gotoPage
      line 75: new AdminisitrablePage
      line 76: page.clickAdministerPage
5. RegisterIT
   Test suite total: 28 a11y checks
   These are parameterizedTests, which take two parameters. The first one is transparent concerning accessibility validation, the second one changes some of the loaded PageElements.

   Each test will play three times, and both cases (true/false) are covered in these situations. Since the cache is in place, we don't mind at all the third repetition, no a11y validaiton will run in it.
   These tests might be migrated to some page tests at one point, since they for the most part check minimal changes in the values in the form.

   1. RegisterJohnSmith
      line 87 (setup):
      line 175 (this.getRegistrationPage(isModal) )
      isModal = true:
      line 205 (RegistrationModal.gotoPage)
      line 23: UsersAdministrationSectionPage.gotoPage -> 2 checks
      line 25: clickAddNewUser
      isModal = false:
      line 207 (RegistrationPage.gotoPage)
      line 42: new RegistrationPage
   2. registerExistingUser
      SAME (4 checks)
   3. registerPasswordTooShort
      SAME (4 checks)
   4. registerDifferentPasswords
      SAME (4 checks)
   5. registerEmptyPassword
      SAME (4 checks)
   6. registerEmptyUserName
      SAME (4 checks)
   7. registerInvalidEmail
      SAME (4 checks)
6. TipsPanelIT
   Test suite total: 4 a11y checks
   1. VerifyTipsParameterIsRestricted
      line 80: gotoPage
      line 81: new PanelViewPage
   2. verifyTipsContentIsExecutedWithTheRightAuthor
      line 114: gotoPage
      line 115: new PanelViewPage
7. AttachmentGalleryPickerMacroIT
   Test suite total: 2 a11y checks
   1. AttachmentGalleryPickerMacro
      line 71: setup.createPage
      line 76: new AttachmentsViewPage
8. CommentsIT
   Test suite total: 4 a11y checks
   1. CommentAsGuest
      line 69: setup.createPage
   2. commentAsLoggedInUser
      line 91: setup.createPage
   3. commentAsAdmin
      line 145: setup.createPage
      line 157: setup.gotoPage
9. UserProfileIT
   Test suite total: 65 a11y checks
   @BeforeEach: 1 check
   line 117: new EditPage
   1. editProfile
      line 125: ProfileUserProfilePage.gotoPage -> 2 checks
      line 126: userProfilePage.editProfile
      line 136: profileEditPage.clickSaveAndView
      line 138: new ProfileUserProfilePage
   2. ChangeAvatarImage
      line 156: ProfileUserProfilePage.gotoPage -> 2 checks
      line 157: userProfilePage.changeAvatarImage
   3. ChangeUserProfile
      line 169: ProfileUserProfilePage.gotoPage -> 2 checks
      line 170: userProfilePage.switchToPreferences
      line 185: preferencesPage.editPreferences
      line 187: preferencesEditPage.clickSaveAndView
      line 189: new ProfileUserProfilePage
      line 190: userProfilePage.switchToPreferences
   4. ChangeDefaultEditor
      line 199: ProfileUserProfilePage.gotoPage -> 2 checks
      line 200: userProfilePage.switchToPreferences
      line 203: preferencesPage.editPreferences
      line 205: preferencesEditPage.clickSaveAndView
      line 207: new ProfileUserProfilePage
      line 208: userProfilePage.switchToPreferences
      line 212: ProfileUserProfilePage.gotoPage -> 2 checks
      line 213: userProfilePage.switchToPreferences
      line 214: preferencesPage.editPreferences
      line 216: preferencesEditPage.clickSaveAndView
      line 218: new ProfileUserProfilePage
      line 219: userProfilePage.switchToPreferences
      line 223: ProfileUserProfilePage.gotoPage -> 2 checks
      line 224: userProfilePage.switchToPreferences
      line 225: preferencesPage.editPreferences
      line 227: preferencesEditPage.clickSaveAndView
      line 229: new ProfileUserProfilePage
      line 230: userProfilePage.switchToPreferences
   5. CommentDoesntOverrideAboutInformation
      line 242: ProfileUserProfilePage.gotoPage -> 2 checks
   6. EnsureDashboardUIAddAnObjectAtFirstEdit
      line 259: ProfileUserProfilePage.gotoPage -> 2 checks
   7. VerifyGroupTab
      line 269: GroupsUserProfilePage.gotoPage
   8. ToggleEnableDisable
      line 283: ProfileUserProfilePage.gotoPage -> 2 checks
      line 290: ProfileUserProfilePage.gotoPage -> 2 checks
      line 300: ProfileUserProfilePage.gotoPage -> 2 checks
      line 309: ProfileUserProfilePage.gotoPage -> 2 checks
   9. disabledUserTest
      line 320: ProfileUserProfilePage.gotoPage -> 2 checks
      line 332: setup.gotoPage
10. NotificationsIT
    Test suite total: 84 a11y checks
    @BeforeAll: 4 checks
    line 102: NotificationsUserProfilePage.gotoPage
    line 106: new NotificationsUserProfilePage
    line 112: NotificationsUserProfilePage.gotoPage
    line 116: new NotificationsUserProfilePage
    1. simpleNotifications
       line 137: setup.createPage
       line 140: setup.gotoPage
       line 142: new NotificationsTrayPage
       line 146: NotificationsUserProfilePage.gotoPage
       line 154: setup.createPage -> loop * 19
       line 156: setup.createPage
       line 160: setup.gotoPage
       line 163: new NotificationsTrayPage
       line 175: setup.gotoPage
       line 178: new NotificationsTrayPage
       line 190: new NotificationsTrayPage
       line 195: NotificationsUserProfilePage.gotoPage
       line 205: setup.gotoPage
       line 208: new NotificationsTrayPage
    2. compositeNotifications
       line 221: NotificationsUserProfilePage.gotoPage
       line 229: setup.gotoPage
       line 230: new NotificationsTrayPage
       line 235: setup.createPage
       line 237: setup.gotoPage
       line 239: new WikiEditPage
       line 242: new ViewPage
       line 244: new WikiEditPage
       line 247: setup.gotoPage
       line 253: setup.gotoPage
       line 255: new NotificationsTrayPage
    3. notificationDisplayerClass
       line 293: setup.gotoPage
       line 294: setup.createPage
       line 297: setup.createPage
       line 307: setup.editObjects
       line 319: NotificationsUserProfilePage.gotoPage
       line 327: setup.gotoPage
       line 329: new WikiEditPage
       line 335: setup.gotoPage
       line 339: new NotificationsTrayPage
    4. ownEventNotifications
       line 357: NotificationsUserProfilePage.gotoPage
       line 364: setup.createPage
       line 367: new NotificationsTrayPage
       line 374: NotificationsUserProfilePage.gotoPage
       line 379: setup.createPage
       line 380: setup.gotoPage
       line 381: new NotificationsTrayPage
    5. guestUsersDontSeeNotificationMenu
       line 402: setup.gotoPage
       line 403: new NotificationsTrayPage
       line 405: new NotificationsTrayPage

All the totals are calculated without taking the cache into account. Some of these checks will be short-wired by an already existing cache entry.

The first thing to consider is the relatively slow WCAG validation time. For each base page such a validation takes between 2 and 6 seconds (axe-core has a timeout of 30 seconds that can be reached on very large pages).

This validation time makes the integration tests significantly slower: on this relatively small test suite, it added up to almost two minutes.

However, there must be solutions to this issue https://www.deque.com/axe/core-documentation/api-documentation/#section-4-performance . Moreover, it's possible to restrict the content analyzed for each base element, making it faster overall.

The reports from axe contain plenty of information, that can be filtered and organized to fulfill our needs (https://github.com/dequelabs/axe-core-maven-html/blob/develop/utilities/src/main/java/com/deque/html/axecore/results/AxeResults.java). In this PoC, only the list of Rules violated are kept and concatenated https://github.com/dequelabs/axe-core-maven-html/blob/develop/utilities/src/main/java/com/deque/html/axecore/results/Rule.java .

Testing it out

The source code for this PoC is available here: https://github.com/Sereza7/xwiki-platform/tree/XWIKI-20541

xwiki-platform-test-docker and xwiki-platform-test-ui should be built.

In order to test it out, you can use this command in the xwiki-platform-menu-test-docker package:

Removing the first parameter will default to no accessibility validation, this can give you an idea of the duration of the test suite.