Audit process

This page is the process to audit the macros against the usability criteria, in two phases: first audit each macro, then report the KOs so that a JIRA ticket covers each of them. The audit of a macro is recorded on its page of the per macro analysis, and the per macro analysis compiles all the audits into progress, history and results.

Where things are

Audit a macro

1. Set up the audit

The audit instance: audit on a local XWiki 18.8.0. Download xwiki-platform-distribution-flavor-jetty-hsqldb-18.8.0.zip from maven.xwiki.org, unzip it, run start_xwiki.sh, and log in as Admin with the password admin: XWiki Standard is already installed. Install the macro first when it is not part of XWiki Standard. Make the wiki multilingual in the administration, under Localization, with French and German as other languages: D4 and the translation check need them. The tools find the instance through XWIKI_AUDIT_URL, see Tools.

With the tools: run audit_macro.py <macroId> to see what it would do, then audit_macro.py <macroId> --apply. It adds the macro object with the criteria version 15.20 and the XWiki version 18.8.0, one parameter object per parameter of the macro descriptor plus one for $content, and the audit display at the top of the macro page. It reads the parameters from the audit instance: for a macro that is not installed there, list them with --parameters a,b,c.

By hand: in the object editor of the macro page, add:

• one MacroAuditClass object, with the macro ID, the status Not started, the criteria version 15.20 and the XWiki version 18.8.0,
• one MacroParameterAuditClass object per parameter of the macro, with the parameter ID, its name in the Edit Macro dialog and its position in the descriptor,
• one more MacroParameterAuditClass object with the parameter ID $content, when the macro has content. Set it to N/A for C3, C5, C7, C8, C9 and C13, the criteria that do not apply to the content.

Then add this line at the top of the page content:

{{include reference="Proposal.WYSIWYGusabilityofXWikimacros.Code.MacroAuditSheet"/}}

When you start the audit, open Edit the audit, set the status to In progress, and fill the auditor.

2. Classify the macro

Classify the macro as a wrapper or a generator, following the Classification of the criteria, and record it in Type. When the macro sits in between, write why in Classification reason.

The type decides which criteria apply: I3, I4, E5 and E6 only apply to wrappers, I5 only applies to generators. Record N/A for the criteria that do not apply. A macro with use cases of both types, like tree, whose static tree is a wrapper and dynamic tree a generator, is Special: write in Classification reason how to tell the use cases apart, and check each criterion only on the use cases where it makes sense, the wrapper criteria on the wrapper use and I5 on the generator use.

3. Gather the sources

Before checking the criteria, gather where the information about the macro is. With the tools: run check_docs.py <macroId> and check_translations.py <macroId>. The first compares the documentation with the macro descriptor and renders its examples, the second lists the texts of the macro in every language of the instance and flags the missing translations. Keep their output for steps 5 and 6.

The Sources table of the audit gives the sources of the macro: the ones that can be computed are links, the documentation, the source code and the other sources are recorded in the audit. audit_macro.py fills the documentation from the Documentation status page; for the others, the table links to a search.

Record anything else specific to the macro in Other sources, one list item each: forum threads, design pages, the macro tests, release notes, the equivalent macro of another product. Write what each one gives, like * [[Forum thread>>https://forum.xwiki.org/t/...]]: why the depth parameter was added.

These sources are the same for every macro, so the audits do not list them:

• The editor: a page of the local XWiki 18.8.0 instance, see Set up the audit. It shows the real insertion, dialogs, rendering and error messages, for D4, I1 to I5, C3 to C7 and E1 to E6.
• The Cross-macro findings: the technical limitations already recorded, and their workarounds.
• The Known issues: the issues reported before the audit, sorted by macro. The audit of a macro lists its own at the bottom, and known_issues.py adds the issues filed since, see Report the KOs.

4. Read the macro descriptor

Many criteria can be checked from the macro descriptor, the metadata the editor builds the Edit Macro dialog from. probe_macro.py of step 6 starts its report with it; by hand, open these URLs on the local XWiki 18.8.0 instance, replacing <macroId>:

• the descriptor: /xwiki/bin/get/Main/WebHome?outputSyntax=plain&sheet=XWiki.WYSIWYG.MacroService&data=descriptor&macroId=<macroId>&syntaxId=xwiki/2.1
• the list of macros with their categories: /xwiki/bin/get/Main/WebHome?outputSyntax=plain&sheet=XWiki.WYSIWYG.MacroService&data=list&syntaxId=xwiki/2.1

5. Pick the reference examples

Several criteria compare the macro with something. Pick these first and record them:

• Compared macros: at least two macros of the same type, for L1 and D3. Record their IDs in Compared macros.
• Most common example: the example of the documentation page for the most common use, checked by L3. C2, C3, C5, C6, C8, E1, E3 and E4 are measured with it. When the documentation has no such example, L3 is KO: write the example you use in the notes, so that the other criteria are checked against the same one. check_docs.py lists the examples of the documentation and says whether each one renders without error.
• Synonyms: the words a user could type to find the macro, other than its name, like "note" or "callout" for warning. The auditor picks them; whether each one is worth supporting is checked when filing the D4 ticket. Write them in the D4 note: D4 is measured with them.

6. Check each criterion

Check the macro against each criterion of the Audit criteria, the copy of the criteria picked for this audit. The Measure column of each criterion says how to check it.

With the tools: run probe_macro.py <macroId> --example "<most common example>" --synonyms <synonyms>. It drives the editor of the audit instance and writes a report with screenshots in probes/<macroId>:

The tools give evidence, not results: read each section against the measure of the criterion and decide. L1, C1 wording, C5 grouping, C6 and C9 stay judgments.

Decisions on the measures

Decisions taken while auditing, which apply to every macro:

• D1: a developer or generic name, like "Container", is a KO when the documentation only shows one use for it, like columns.
• D1, D3: a name or a category that leads users to another feature, like "Comment" for the page comments, is a KO. A developer widget that Quick Actions lists before any entry doing what the user searched for, like Attachment Picker for "attachment", is a minor KO for D1.
• D2: the description says what the macro produces. Repeating the name, like "Displays a warning message note.", is a KO, even for a simple wrapper. An output that does more than the description says, like the attachments listed by children, is a KO too. A description identical to the one of another macro, like include and display, is a minor KO, and a KO for the variant when it does not say what sets it apart, like display, which renders the page in its own context. A description that does not state a rule the user needs to insert the macro, like the single list of footnotes of putFootnotes, whose insertion is otherwise lost silently, is a KO.
• D3: a macro can have several categories since XWiki 14.6. A category where users would also look for the macro, like Development for code, is a minor KO when it is missing.
• D3: a macro in a user category, like Content, whose parameters need developer knowledge, like liveData, is a minor KO; whether the parameters or the category should change is discussed with the community. Until then, the macro is audited as a user macro.
• D4: only the Quick Actions menu counts, for the English name, ID and synonyms, and for the name in at least one other language. A translated name equal to the English one, like the German "Box", is OK, as long as a language with a different name finds the macro. A typo in a translation is fixed on l10n.xwiki.org, it is not recorded in the audit. The Insert Macro dialog does not search macro IDs, which is generic, see XWIKI-20230.
• D4 for a technical macro: a macro named after its technology, like html, is OK even when few synonyms find it.
• Missing translations: a technical macro of a community supported extension that is not translated at all, like jira, is accepted: the missing translations are kept as a note, not counted.
• Synonyms of D4: the auditor picks them; whether each one is worth supporting is decided when filing the ticket.
• L1: a difference with the compared macros that is by design is OK: macros of the same type do not need the same features. A content format specific to one macro, like the image list of gallery, is a KO.
• L2: parameters that are documented only on a legacy extensions.xwiki.org page, not in the new documentation, are a KO.
• L2: no example of the most common call, like office whose every example sets filterStyles="false", is a KO.
• L3: an example that needs a setup step the documentation states, like the office server, is OK.
• L3: an example that needs an extension missing from XWiki Standard, like a Confluence syntax, is a KO: users are not assumed to have any extension.
• L3: no documentation example showing the most common use is a KO; examples that only show off a parameter are a minor KO.
• L3: a documented example with a typo, like the unclosed {{dashboard}} of dashboard, is a minor KO. So is an example the editor cannot reproduce, like a page that the page picker hides.
• I1: a companion macro inserted silently, like putFootnotes with footnote, is a KO when nothing in the editor explains it.
• I1 for a generator: the reference is the children macro.
• I1: two Quick Actions entries for the same macro, like the icon picker and the generic Icon entry of displayIcon, is a KO: the generic entry should not be listed when a dedicated one does the same thing.
• I2: a result that shows nothing in view mode, like an empty box, is a KO: the user expects to see what they inserted. A message that explains an empty result, shown as information and not as an error, like "No pages found" for children, is OK, and so is an empty result for which the editor shows a placeholder saying what it is. A mandatory content is a KO when an empty content would give a sensible result, like an empty message box.
• I2: an Edit Macro dialog that refuses an empty mandatory value without a message is OK, with a note: it is the generic behavior of the dialog.
• I2: an empty result should show a minimal wrapping, like the frame of an empty dashboard. A dedicated placeholder that only shows after reopening the page, or that does not say the result is empty, like the one of tree, is a minor KO.
• I4: a default content inserted as real, unselected text is a minor KO.
• I5: a macro with no mandatory parameter is OK.
• E1: a generator that only shows the generic "macro:<id>" placeholder in the editor is a KO: it should at least show a placeholder that says what it generates. A macro that renders with JavaScript or asynchronously, or whose output is too heavy for the editor, like liveData, jira or office, is a KO when the editor shows a loading spinner, nothing or a generic placeholder: it should show an edit-mode replacement that says what the macro shows.
• E2: a value accepted silently is a KO; the note gives its source. A reference to a page that does not exist yet is a valid use, not a silent acceptance. This only holds for a link target: a missing page the macro reads its data from, like the reference of tree, is a KO, since the editor could show that the page does not exist and propose to create it.
• E3: a block macro that renders inline in a list item or a table cell, and stays usable, is OK with a note.
• E4: inline support that only works for some values of a parameter, like the enum style of jira, is a minor KO: inline, the parameter should only offer the values that work.
• E5: a macro inserted with no content area, as a placeholder that never receives the cursor, is a KO for E5 too, even when it is already a KO for I4. So is a content area that is visible but never receives the cursor, like the empty box of code. A click target smaller than 24 by 24 pixels, the minimum of WCAG 2.5.8, like the number of an empty footnote, is a minor KO.
• Development macros: a macro in the Development category, like attachmentGalleryPicker, is audited as a developer macro: technical terms are OK for C1. A description that does not say what the parameter is for, like "The id of the attachment picker macro", is still a KO.
• C1: a technical name, like "Scope", is OK when the description explains it. Technical terms are OK for a styling parameter, like cssClass, which another wording would only make more confusing. Other developer terms, like "Objects" in documentTree, are a KO when the macro targets every user.
• C1: a description that uses a wrong term, like "message box" for the box macro, is a KO. An empty value with a special meaning, like the automatic detection of the code language, that the description does not explain is a minor KO.
• C1: a parameter whose name or behavior differs from the parameter other macros use for the same purpose, like class of tree and style of dashboard next to cssClass, is a KO.
• C3: a field that preselects the most used value, like the first option of a select, is OK even when no default is declared. A select that opens on "-", or an empty field, that does not show which value it stands for is a KO, minor when no single value is used in most cases. An empty field whose description gives a wrong default, like the columns of documents, is a KO. An almost mandatory parameter, like the source of liveData, may open empty when it comes first in the dialog. A value specific to each wiki, like the URL of jira, is a KO when no administration setting gives a default.
• C4: a plain text field for a number, like a heading level, is a KO. A missing picker stays a KO even when a proper field needs design work. A plain text field for a CSS length, like a width, is OK: a CSS value picker is listed in the Cross-macro findings.
• C4 on $content: a dialog text area for content edited in place is counted under E6 only.
• C5: each parameter of a group that is too large gets the KO; they are grouped when the ticket is filed. A group of more than 7 fields, content included, is a minor KO, and of more than 9 a KO.
• C6: when the most common example sets no parameter, the order is OK. Only the common parameters that are hard to reach get the KO, like source and properties of liveData; the parameters above them are OK, since they move down once the first are fixed.
• Hidden and deprecated parameters: C9 is OK and the dialog criteria are N/A, except for a parameter that a documented example sets, which is rated like a visible one. A parameter documented as deprecated but shown in the dialog, like space and parent of documents, is a C9 KO, since hiding it is the fix; its C4 and C8 are OK with a note.
• C7: deprecated alternatives of a parameter shown as separate fields, like space and parent next to location in documents, are a KO on each of them.
• C7: a parameter that replaces the content, like source for code, is OK with a note: only parameters can be part of a feature, a limitation listed in the Cross-macro findings.
• C8: a rarely used parameter, like cssClass, that is not marked as advanced is a KO, even when its technical wording is OK for C1. So is a parameter that does not change what the macro does, like filterStyles of office, even when the documentation examples set it.
• C8, C9: a parameter with a single value in XWiki Standard that customizations can extend, like layout and style of dashboard, is not hidden: C9 OK, and C8 KO when it is not advanced.
• C9: a parameter with a realistic use is OK, even when no documented example sets it. An inherited parameter that users would not expect for the macro, like the image of code, is a KO: it should be hidden.
• Special macros: a parameter whose format or meaning changes with the use case, like the node ids of tree, is a KO for C4 and C13 when it has no picker or declared format. Note it as a hint that the use cases could be split into two macros.
• Bugs outside the criteria: a bug of the macro that no criterion covers, like the cssClass of container removing its columns, is filed in JIRA and kept as a note, not counted as a KO.
• Generic bugs: a bug of every macro, of the editor or of the Edit Macro dialog is not recorded in the audit of a macro. For example, the Edit Macro dialog does not enforce the declared constraints of C13, see XWIKI-25287.

Record each result in Edit the audit. The criteria of the Configuration section have one result per parameter, in the Parameters table of the form.

7. Record the results

• Record OK, KO or N/A for every criterion. Leave no criterion empty in a reviewed audit: the results count empty values as Not set.
• Do not look for tickets yet: the KOs and tickets table at the bottom of the audit lists every KO, and the tickets are linked when you report the KOs.
• Write the notes as one paragraph per criterion, starting with its ID, like "D4: the macro name gives no result.": the audit shows each one on the row of its criterion. A paragraph that starts with no criterion ID is shown under Other notes. Parameter notes go in the notes of the parameter, in the Parameters table. For each KO, give an example that reproduces it, so that the ticket can be filed from the note alone: for E2, the source of an invalid value the macro accepts silently, like {{warning width="abc"}}Text{{/warning}} . Write "minor" after the ID of a KO with a small impact.
• When a criterion cannot be met because of a technical limitation, record the limitation and the chosen workaround once in the Cross-macro findings, link it from the notes of every affected macro, and mark the criterion as KO. Apply the same workaround to every affected macro.

8. Mark the macro as reviewed

Set the status to Reviewed, even when every criterion is OK, so that "no issue" never means "not reviewed yet". Fill the Audit date: the history of the Per macro analysis groups the reviewed macros by this date, and uses the last change of the macro page when it is empty.

Then delete the earlier analysis: the content written on the macro page before the audit, below the audit display. The audit replaces it, so keep only the include line at the top of the page, either by editing the page in the wiki editor or with audit_macro.py <macroId> --delete-earlier-analysis --apply. The deleted content stays in the page history. The tool also deletes the pictures the earlier analysis showed, screenshots attached to the proposal home page or to the macro page, when no other page uses them; by hand, delete them from the attachments of those pages. Until it is deleted, the audit shows a warning, and the Per macro analysis lists the macro.

The macro then counts in the progress, the history and the results of the Per macro analysis.

Report the KOs

Once the main batch of macros is reviewed, make sure that a JIRA ticket covers every KO. The KOs not reported yet table of the Per macro analysis lists every KO of the reviewed macros that no ticket covers yet, with the known issues to check: report from it, so that the KOs of several macros can be grouped in one ticket. The KOs and tickets table at the bottom of each audit shows the same KOs for one macro, with the tickets that already cover them. A macro can also be reported on its own as soon as it is reviewed, from its audit.

1. Refresh the known issues. Run known_issues.py --apply, then sort the issues it lists as To sort in the object editor of the Known issues page, so that a ticket filed since the last search is not duplicated. Doing it once at the start of a reporting session is enough.
2. Link the existing tickets. For each KO of the table, check the known issues suggested in italics: the ones filed for the macro, alone or with a few other macros, under the same criterion. The issues of every macro are not suggested, since they repeat on every audit. When one covers the KO, use Add a ticket on its row: it adds a ticket covering this KO to the audit of the macro and opens its form. Fill the JIRA key, set the origin to Existing, and add the other KOs of the macro the ticket covers. When the ticket only fixes part of a KO, list that KO under KOs partly covered instead: the ticket shows on its row, and the KO stays in the table until another ticket covers the rest. When several partial tickets fix the whole KO between them, add the KO to KOs covered together in the audit of the macro: it then counts as covered.
3. File the new tickets. Group the KOs left, see below, and create one JIRA ticket per group. Then record it with Add a ticket on the row of one of its KOs, with the origin New, and add the other KOs of the macro it covers. When a ticket covers KOs of several macros, record it once on each of them. Before filing, check the Dependencies of the new tickets on the Triage page: when a KO of the new ticket is listed there, link the new ticket in JIRA as depending on the existing ticket of that row.
4. Check the coverage. A KO leaves the table as soon as a ticket recorded on its macro lists it under KOs covered. The reporting is done when the table says that every KO is covered.

One ticket can cover several KOs, so there is no need for one ticket per KO. Group the KOs that one change would fix: the same criterion on several parameters, like all the C4 KOs of the macro, several criteria on one parameter, or the same problem in several macros, which the table shows side by side. A ticket about a problem shared by several macros is recorded on the page of each of them. A KO caused by a technical limitation is covered by the ticket of its cross-macro findings entry.

A KO is written as the criterion ID for a macro criterion, like D4, and as the criterion ID and the parameter ID for a configuration criterion, like C4:title or C1:$content.

Tools

The tools are Python scripts that automate the setup of an audit and the maintenance of the audit pages. Download macro-audit-tools.zip and unzip it. Its README.md gives the details.

• Requirements: Python 3.9 or later, and a design.xwiki.org account with edit rights on this proposal. Publishing the pages with analysis_page.py also needs script rights. probe_macro.py also needs Playwright: pip install playwright then playwright install chromium.
• Audit instance: the checks read the local XWiki 18.8.0 at XWIKI_AUDIT_URL, http://localhost:8080/xwiki by default, as XWIKI_AUDIT_USER with XWIKI_AUDIT_PASSWORD, Admin and admin by default. They write their test pages in its MacroAudit space.
• Credentials: set the XWIKI_USER and XWIKI_PASSWORD environment variables, or write them as KEY=value lines in a ~/.xwiki-credentials file.
• Known issues: known_issues.py --apply runs the JIRA searches again, refreshes the title and status of every known issue, and adds the new ones as To sort. It never changes how an issue was sorted.
• Dry run first: every script only prints what it would change. Add --apply to write. Every write is a minor revision, so page watchers are not notified.

The archive is updated each time analysis_page.py --apply publishes this page, so it always holds the tools that produced the published pages.

Maintain the audit

• The criteria change: the audit keeps using the copy of version 15.20. To move to a new version, run sync.py then sync.py --apply to update the classes, and change CRITERIA_VERSION in analysis_page.py then run analysis_page.py --apply to take a new copy. A criterion removed from the criteria page is disabled in the classes, not deleted, so the results already recorded are kept. Audits keep the criteria version they were done against.
• The statistics, the charts or the audit display change: edit MacroAuditStats.xwiki or MacroAuditSheet.xwiki, then run analysis_page.py --apply.
• The searches for known issues change: edit QUERIES in known_issues.py, then run it. The issues found only by a removed search stay on the page.