Audit process

This page is the process to audit the macros against the usability criteria, in two phases: first audit each macro, then report the KOs so that a JIRA ticket covers each of them. The audit of a macro is recorded on its page of the per macro analysis, and the Usability analysis compiles all the audits into progress, history and results.

Where things are

Audit a macro

1. Set up the audit

The audit instance: audit on a local XWiki 18.8.0. Download xwiki-platform-distribution-flavor-jetty-hsqldb-18.8.0.zip from maven.xwiki.org, unzip it, run start_xwiki.sh, and log in as Admin with the password admin: XWiki Standard is already installed. Install the macro first when it is not part of XWiki Standard. Make the wiki multilingual in the administration, under Localization, with French and German as other languages: D4 and the translation check need them. The tools find the instance through XWIKI_AUDIT_URL, see Tools.

With the tools: run audit_macro.py <macroId> to see what it would do, then audit_macro.py <macroId> --apply. It adds the macro object with the criteria version 15.17 and the XWiki version 18.8.0, one parameter object per parameter of the macro descriptor plus one for $content, and the audit display at the top of the macro page. It reads the parameters from the audit instance: for a macro that is not installed there, list them with --parameters a,b,c.

By hand: in the object editor of the macro page, add:

• one MacroAuditClass object, with the macro ID, the status Not started, the criteria version 15.17 and the XWiki version 18.8.0,
• one MacroParameterAuditClass object per parameter of the macro, with the parameter ID, its name in the Edit Macro dialog and its position in the descriptor,
• one more MacroParameterAuditClass object with the parameter ID $content, when the macro has content. Set it to N/A for C3, C5, C7, C8, C9 and C13, the criteria that do not apply to the content.

Then add this line at the top of the page content:

{{include reference="Proposal.WYSIWYGusabilityofXWikimacros.Code.MacroAuditSheet"/}}

When you start the audit, open Edit the audit, set the status to In progress, and fill the auditor.

2. Classify the macro

Classify the macro as a wrapper or a generator, following the Classification of the criteria, and record it in Type. When the macro sits in between, write why in Classification reason.

The type decides which criteria apply: I3, I4, E5 and E6 only apply to wrappers, I5 only applies to generators. Record N/A for the criteria that do not apply.

3. Gather the sources

Before checking the criteria, gather where the information about the macro is. With the tools: run check_docs.py <macroId> and check_translations.py <macroId>. The first compares the documentation with the macro descriptor and renders its examples, the second lists the texts of the macro in every language of the instance and flags the missing translations. Keep their output for steps 5 and 6.

The Sources table of the audit gives the sources of the macro: the ones that can be computed are links, the documentation, the source code and the other sources are recorded in the audit. audit_macro.py fills the documentation from the Documentation status page; for the others, the table links to a search.

Record anything else specific to the macro in Other sources, one list item each: forum threads, design pages, the macro tests, release notes, the equivalent macro of another product. Write what each one gives, like * [[Forum thread>>https://forum.xwiki.org/t/...]]: why the depth parameter was added.

These sources are the same for every macro, so the audits do not list them:

• The editor: a page of the local XWiki 18.8.0 instance, see Set up the audit. It shows the real insertion, dialogs, rendering and error messages, for D4, I1 to I5, C3 to C7 and E1 to E6.
• The Cross-macro findings: the technical limitations already recorded, and their workarounds.
• The Known issues: the issues reported before the audit, sorted by macro. The audit of a macro lists its own at the bottom, and known_issues.py adds the issues filed since, see Report the KOs.

4. Read the macro descriptor

Many criteria can be checked from the macro descriptor, the metadata the editor builds the Edit Macro dialog from. probe_macro.py of step 6 starts its report with it; by hand, open these URLs on the local XWiki 18.8.0 instance, replacing <macroId>:

• the descriptor: /xwiki/bin/get/Main/WebHome?outputSyntax=plain&sheet=XWiki.WYSIWYG.MacroService&data=descriptor&macroId=<macroId>&syntaxId=xwiki/2.1
• the list of macros with their categories: /xwiki/bin/get/Main/WebHome?outputSyntax=plain&sheet=XWiki.WYSIWYG.MacroService&data=list&syntaxId=xwiki/2.1

5. Pick the reference examples

Several criteria compare the macro with something. Pick these first and record them:

• Compared macros: at least two macros of the same type, for L1 and D3. Record their IDs in Compared macros.
• Most common example: the example of the documentation page for the most common use, checked by L3. C2, C3, C5, C6, C8, E1, E3 and E4 are measured with it. When the documentation has no such example, L3 is KO: write the example you use in the notes, so that the other criteria are checked against the same one. check_docs.py lists the examples of the documentation and says whether each one renders without error.
• Synonyms: the words a user could type to find the macro, other than its name, like "note" or "callout" for warning. Write them in the D4 note: D4 is measured with them.

6. Check each criterion

Check the macro against each criterion of the Criteria section of the Usability analysis, the copy of the criteria picked for this audit. The Measure column of each criterion says how to check it.

With the tools: run probe_macro.py <macroId> --example "<most common example>" --synonyms <synonyms>. It drives the editor of the audit instance and writes a report with screenshots in probes/<macroId>:

The tools give evidence, not results: read each section against the measure of the criterion and decide. L1, C1 wording, C5 grouping, C6 and C9 stay judgments.

Record each result in Edit the audit. The criteria of the Configuration section have one result per parameter, in the Parameters table of the form.

7. Record the results

• Record OK, KO or N/A for every criterion. Leave no criterion empty in a reviewed audit: the results count empty values as Not set.
• Do not look for tickets yet: the KOs and tickets table at the bottom of the audit lists every KO, and the tickets are linked when you report the KOs.
• Write the notes as one paragraph per criterion, starting with its ID, like "C4: the reference field has no picker."
• When a criterion cannot be met because of a technical limitation, record the limitation and the chosen workaround once in the Cross-macro findings, link it from the notes of every affected macro, and mark the criterion as KO. Apply the same workaround to every affected macro.

8. Mark the macro as reviewed

Set the status to Reviewed, even when every criterion is OK, so that "no issue" never means "not reviewed yet". Fill the Audit date: the history of the Usability analysis groups the reviewed macros by this date, and uses the last change of the macro page when it is empty.

Then delete the earlier analysis: the content written on the macro page before the audit, below the audit display. The audit replaces it, so keep only the include line at the top of the page, either by editing the page in the wiki editor or with audit_macro.py <macroId> --delete-earlier-analysis --apply. The deleted content stays in the page history. Until it is deleted, the audit shows a warning, and the Usability analysis lists the macro.

The macro then counts in the progress, the history and the results of the Usability analysis.

Report the KOs

Once the main batch of macros is reviewed, make sure that a JIRA ticket covers every KO. The KOs not reported yet table of the Usability analysis lists every KO of the reviewed macros that no ticket covers yet, with the known issues to check: report from it, so that the KOs of several macros can be grouped in one ticket. The KOs and tickets table at the bottom of each audit shows the same KOs for one macro, with the tickets that already cover them. A macro can also be reported on its own as soon as it is reviewed, from its audit.

1. Refresh the known issues. Run known_issues.py --apply, then sort the issues it lists as To sort in the object editor of the Known issues page, so that a ticket filed since the last search is not duplicated. Doing it once at the start of a reporting session is enough.
2. Link the existing tickets. For each KO of the table, check the known issues suggested in italics: the ones of the macro under the same criterion, and the number of issues of every macro under that criterion, linked to the Known issues page. When one covers the KO, use Add a ticket on its row: it adds a ticket covering this KO to the audit of the macro and opens its form. Fill the JIRA key, set the origin to Existing, and add the other KOs of the macro the ticket covers.
3. File the new tickets. Group the KOs left, see below, and create one JIRA ticket per group. Then record it with Add a ticket on the row of one of its KOs, with the origin New, and add the other KOs of the macro it covers. When a ticket covers KOs of several macros, record it once on each of them.
4. Check the coverage. A KO leaves the table as soon as a ticket recorded on its macro covers it. The reporting is done when the table says that every KO is covered.

One ticket can cover several KOs, so there is no need for one ticket per KO. Group the KOs that one change would fix: the same criterion on several parameters, like all the C4 KOs of the macro, several criteria on one parameter, or the same problem in several macros, which the table shows side by side. A ticket about a problem shared by several macros is recorded on the page of each of them. A KO caused by a technical limitation is covered by the ticket of its cross-macro findings entry.

A KO is written as the criterion ID for a macro criterion, like D4, and as the criterion ID and the parameter ID for a configuration criterion, like C4:title or C1:$content.

Tools

The tools are Python scripts that automate the setup of an audit and the maintenance of the audit pages. Download macro-audit-tools.zip and unzip it. Its README.md gives the details.

• Requirements: Python 3.9 or later, and a design.xwiki.org account with edit rights on this proposal. Publishing the pages with analysis_page.py also needs script rights. probe_macro.py also needs Playwright: pip install playwright then playwright install chromium.
• Audit instance: the checks read the local XWiki 18.8.0 at XWIKI_AUDIT_URL, http://localhost:8080/xwiki by default, as XWIKI_AUDIT_USER with XWIKI_AUDIT_PASSWORD, Admin and admin by default. They write their test pages in its MacroAudit space.
• Credentials: set the XWIKI_USER and XWIKI_PASSWORD environment variables, or write them as KEY=value lines in a ~/.xwiki-credentials file.
• Known issues: known_issues.py --apply runs the JIRA searches again, refreshes the title and status of every known issue, and adds the new ones as To sort. It never changes how an issue was sorted.
• Dry run first: every script only prints what it would change. Add --apply to write. Every write is a minor revision, so page watchers are not notified.

The archive is updated each time analysis_page.py --apply publishes this page, so it always holds the tools that produced the published pages.

Maintain the audit

• The criteria change: the audit keeps using the copy of version 15.17. To move to a new version, run sync.py then sync.py --apply to update the classes, and change CRITERIA_VERSION in analysis_page.py then run analysis_page.py --apply to take a new copy. A criterion removed from the criteria page is disabled in the classes, not deleted, so the results already recorded are kept. Audits keep the criteria version they were done against.
• The statistics, the charts or the audit display change: edit MacroAuditStats.xwiki or MacroAuditSheet.xwiki, then run analysis_page.py --apply.
• The searches for known issues change: edit QUERIES in known_issues.py, then run it. The issues found only by a removed search stay on the page.