Wiki source code of Audit process
Last modified by Lucas Charpentier on 2026/10/13 16:37
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | This page is the process to audit the macros against the usability criteria, in two phases: first [[audit each macro>>||anchor="HAuditamacro"]], then [[report the KOs>>||anchor="HReporttheKOs"]] so that a JIRA ticket covers each of them. The audit of a macro is recorded on its page of the [[per macro analysis>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Macros.WebHome]], and the [[Usability analysis>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.WebHome]] compiles all the audits into progress, history and results. | ||
| 2 | |||
| 3 | == Where things are == | ||
| 4 | |||
| 5 | |=What|=Where | ||
| 6 | |Criteria used by the audit|The [[Criteria section of the Usability analysis>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.WebHome||anchor="HCriteria"]], a copy of version 15.17 of the [[Usability criteria>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Usability criteria.WebHome]] page. Later changes of that page do not apply until the audit moves to a new version. | ||
| 7 | |XWiki version audited|XWiki 18.8.0, the latest release when the audit started. Every macro is audited on this version, so that the results are comparable, even when a later version is released during the audit. design.xwiki.org runs another version: audit on a local instance, see [[Set up the audit>>||anchor="H1.Setuptheaudit"]]. | ||
| 8 | |Audit of a macro|At the top of the macro page, in the [[per macro analysis>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Macros.WebHome]]. **Edit the audit**, under the audit, opens the form. | ||
| 9 | |Progress, history and results|The [[Usability analysis>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.WebHome]]. Everything there is computed from the audits, so there is nothing to update by hand. | ||
| 10 | |JIRA issues reported before the audit|The [[Known issues>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.Known issues.WebHome]], sorted by the macro they affect. | ||
| 11 | |Technical limitations shared by several macros|The [[Cross-macro findings>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Cross-macro findings.WebHome]]. | ||
| 12 | |Audit data|Two classes, in hidden pages: [[MacroAuditClass>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Code.MacroAuditClass]] holds one object per macro, with one field per criterion that is checked once for the macro. [[MacroParameterAuditClass>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Code.MacroParameterAuditClass]] holds one object per parameter of the macro, content included as ##$content##, with one field per Configuration criterion. [[MacroAuditTicketClass>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Code.MacroAuditTicketClass]] holds one object per JIRA ticket covering KOs of the macro. [[KnownIssueClass>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Code.KnownIssueClass]] holds one object per known issue, on the Known issues page. | ||
| 13 | |Display of the audit data|Hidden pages of the ##Proposal.WYSIWYGusabilityofXWikimacros.Code## space: ##MacroAuditSheet## shows the audit on a macro page, ##MacroAuditStats## computes the statistics and charts of the Usability analysis, ##MacroAuditLiveData##, ##MacroParameterAuditLiveData## and ##MacroParameterAuditResults## give the tables of all audit values, ##KnownIssuesJSON## gives the known issues to ##known_issues.py##. | ||
| 14 | |Tools|[[macro-audit-tools.zip>>attach:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.Audit [email protected]]], attached to this page: ##audit_macro.py## sets up the audit of a macro, ##check_docs.py##, ##check_translations.py## and ##probe_macro.py## collect the evidence of the criteria on the audit instance, ##sync.py## updates the classes from the criteria page, ##analysis_page.py## publishes the Usability analysis, this page, the audit display and the statistics page, ##known_issues.py## refreshes the Known issues from JIRA. See [[Tools>>||anchor="HTools"]]. Every step below can also be done by hand. | ||
| 15 | |||
| 16 | == Audit a macro == | ||
| 17 | |||
| 18 | === 1. Set up the audit === | ||
| 19 | |||
| 20 | **The audit instance:** audit on a local XWiki 18.8.0. Download ##xwiki-platform-distribution-flavor-jetty-hsqldb-18.8.0.zip## from [[maven.xwiki.org>>https://maven.xwiki.org/releases/org/xwiki/platform/xwiki-platform-distribution-flavor-jetty-hsqldb/18.8.0/]], unzip it, run ##start_xwiki.sh##, and log in as Admin with the password admin: XWiki Standard is already installed. Install the macro first when it is not part of XWiki Standard. Make the wiki multilingual in the administration, under Localization, with French and German as other languages: D4 and the translation check need them. The tools find the instance through ##XWIKI_AUDIT_URL##, see [[Tools>>||anchor="HTools"]]. | ||
| 21 | |||
| 22 | **With the [[tools>>||anchor="HTools"]]:** run ##audit_macro.py <macroId>## to see what it would do, then ##audit_macro.py <macroId> ~-~-apply##. It adds the macro object with the criteria version 15.17 and the XWiki version 18.8.0, one parameter object per parameter of the macro descriptor plus one for ##$content##, and the audit display at the top of the macro page. It reads the parameters from the audit instance: for a macro that is not installed there, list them with ##~-~-parameters a,b,c##. | ||
| 23 | |||
| 24 | **By hand:** in the object editor of the macro page, add: | ||
| 25 | |||
| 26 | * one ##MacroAuditClass## object, with the macro ID, the status Not started, the criteria version 15.17 and the XWiki version 18.8.0, | ||
| 27 | * one ##MacroParameterAuditClass## object per parameter of the macro, with the parameter ID, its name in the Edit Macro dialog and its position in the descriptor, | ||
| 28 | * one more ##MacroParameterAuditClass## object with the parameter ID ##$content##, when the macro has content. Set it to N/A for C3, C5, C7, C8, C9 and C13, the criteria that do not apply to the content. | ||
| 29 | |||
| 30 | Then add this line at the top of the page content: | ||
| 31 | |||
| 32 | {{code language="none"}} | ||
| 33 | {{include reference="Proposal.WYSIWYGusabilityofXWikimacros.Code.MacroAuditSheet"/}} | ||
| 34 | {{/code}} | ||
| 35 | |||
| 36 | When you start the audit, open **Edit the audit**, set the status to In progress, and fill the auditor. | ||
| 37 | |||
| 38 | === 2. Classify the macro === | ||
| 39 | |||
| 40 | Classify the macro as a wrapper or a generator, following the [[Classification>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.WebHome||anchor="H0.Classification"]] of the criteria, and record it in **Type**. When the macro sits in between, write why in **Classification reason**. | ||
| 41 | |||
| 42 | The type decides which criteria apply: I3, I4, E5 and E6 only apply to wrappers, I5 only applies to generators. Record N/A for the criteria that do not apply. | ||
| 43 | |||
| 44 | === 3. Gather the sources === | ||
| 45 | |||
| 46 | Before checking the criteria, gather where the information about the macro is. **With the [[tools>>||anchor="HTools"]]:** run ##check_docs.py <macroId>## and ##check_translations.py <macroId>##. The first compares the documentation with the macro descriptor and renders its examples, the second lists the texts of the macro in every language of the instance and flags the missing translations. Keep their output for steps 5 and 6. | ||
| 47 | |||
| 48 | The **Sources** table of the audit gives the sources of the macro: the ones that can be computed are links, the documentation, the source code and the other sources are recorded in the audit. ##audit_macro.py## fills the documentation from the [[Documentation status>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Documentation status.WebHome]] page; for the others, the table links to a search. | ||
| 49 | |||
| 50 | |=Source|=Where to find it|=What it gives|=Used for | ||
| 51 | |Documentation|The new documentation tree on www.xwiki.org, or the legacy pages on extensions.xwiki.org. The [[Documentation status>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Documentation status.WebHome]] page says which one each macro has.|The parameters, their default values and the examples, including the most common example.|L2, L3, and the criteria measured with the most common example | ||
| 52 | |Source code|On GitHub: xwiki-rendering for the rendering macros like box or toc, xwiki-platform for the other macros of XWiki Standard, a wiki macro being a page of a XAR module, and xwiki-contrib for most extensions.|How the macro declares its parameters and content: types, groups, features, advanced and hidden flags, constraints, content type, inline support.|C4, C5, C7, C8, C9, C13, E4, E6 | ||
| 53 | |Macro descriptor|The Macro Service of the wiki, see [[Read the macro descriptor>>||anchor="H4.Readthemacrodescriptor"]].|What the Insert Macro and Edit Macro dialogs are built from.|D1 to D3, C1 to C9, E4 | ||
| 54 | |Translations|[[l10n.xwiki.org>>https://l10n.xwiki.org]], or the translation file of the source code.|The name, description and parameter names, in each language.|D1, D2, D4, C1 | ||
| 55 | |Uses in XWiki pages|A GitHub code search for the macro call, like ##~{~{toc##, across every repository of the [[xwiki organization>>https://github.com/xwiki]].|How XWiki itself uses the macro, which gives a realistic example when the documentation has none.|L3, C2, C3, C6, C8, C9 | ||
| 56 | |Compared macros|The audits of the macros recorded in **Compared macros**.|What the macros of the same type do.|L1, D3 | ||
| 57 | |||
| 58 | Record anything else specific to the macro in **Other sources**, one list item each: forum threads, design pages, the macro tests, release notes, the equivalent macro of another product. Write what each one gives, like ##* ~[~[Forum thread>>https:~/~/forum.xwiki.org/t/...]]: why the depth parameter was added##. | ||
| 59 | |||
| 60 | These sources are the same for every macro, so the audits do not list them: | ||
| 61 | |||
| 62 | * **The editor:** a page of the local XWiki 18.8.0 instance, see [[Set up the audit>>||anchor="H1.Setuptheaudit"]]. It shows the real insertion, dialogs, rendering and error messages, for D4, I1 to I5, C3 to C7 and E1 to E6. | ||
| 63 | * **The [[Cross-macro findings>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Cross-macro findings.WebHome]]:** the technical limitations already recorded, and their workarounds. | ||
| 64 | * **The [[Known issues>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.Known issues.WebHome]]:** the issues reported before the audit, sorted by macro. The audit of a macro lists its own at the bottom, and ##known_issues.py## adds the issues filed since, see [[Report the KOs>>||anchor="HReporttheKOs"]]. | ||
| 65 | |||
| 66 | === 4. Read the macro descriptor === | ||
| 67 | |||
| 68 | Many criteria can be checked from the macro descriptor, the metadata the editor builds the Edit Macro dialog from. ##probe_macro.py## of step 6 starts its report with it; by hand, open these URLs on the local XWiki 18.8.0 instance, replacing ##<macroId>##: | ||
| 69 | |||
| 70 | * the descriptor: ##/xwiki/bin/get/Main/WebHome?outputSyntax=plain&sheet=XWiki.WYSIWYG.MacroService&data=descriptor¯oId=<macroId>&syntaxId=xwiki/2.1## | ||
| 71 | * the list of macros with their categories: ##/xwiki/bin/get/Main/WebHome?outputSyntax=plain&sheet=XWiki.WYSIWYG.MacroService&data=list&syntaxId=xwiki/2.1## | ||
| 72 | |||
| 73 | |=Descriptor field|=Used for | ||
| 74 | |##name##, ##description##|D1, D2 | ||
| 75 | |##categories## in the list|D3 | ||
| 76 | |##supportsInlineMode##|E4 | ||
| 77 | |##mandatory## of each parameter|C2, I5 | ||
| 78 | |##defaultValue##|C3 | ||
| 79 | |##displayType##|C4. For the content, ##java.util.List<org.xwiki.rendering.block.Block>## means rich text, which E6 expects for a Java macro. | ||
| 80 | |The ##GROUP## entries, their ##children## and ##feature##|C5, C7 | ||
| 81 | |##order##|C6 | ||
| 82 | |##advanced##|C8 | ||
| 83 | |##hidden##, ##deprecated##|C9 | ||
| 84 | |||
| 85 | === 5. Pick the reference examples === | ||
| 86 | |||
| 87 | Several criteria compare the macro with something. Pick these first and record them: | ||
| 88 | |||
| 89 | * **Compared macros:** at least two macros of the same type, for L1 and D3. Record their IDs in **Compared macros**. | ||
| 90 | * **Most common example:** the example of the documentation page for the most common use, checked by L3. C2, C3, C5, C6, C8, E1, E3 and E4 are measured with it. When the documentation has no such example, L3 is KO: write the example you use in the notes, so that the other criteria are checked against the same one. ##check_docs.py## lists the examples of the documentation and says whether each one renders without error. | ||
| 91 | * **Synonyms:** the words a user could type to find the macro, other than its name, like "note" or "callout" for warning. Write them in the D4 note: D4 is measured with them. | ||
| 92 | |||
| 93 | === 6. Check each criterion === | ||
| 94 | |||
| 95 | Check the macro against each criterion of the [[Criteria section of the Usability analysis>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.WebHome||anchor="HCriteria"]], the copy of the criteria picked for this audit. The Measure column of each criterion says how to check it. | ||
| 96 | |||
| 97 | **With the [[tools>>||anchor="HTools"]]:** run ##probe_macro.py <macroId> ~-~-example "<most common example>" ~-~-synonyms <synonyms>##. It drives the editor of the audit instance and writes a report with screenshots in ##probes/<macroId>##: | ||
| 98 | |||
| 99 | |=Report section|=Evidence for | ||
| 100 | |Descriptor|D1, D2, D3, C2, C3, C5 to C9, C13, E4 | ||
| 101 | |D4|D4: rank in the Quick Actions menu and the Insert Macro dialog, for the name, the ID, the localized names and the synonyms | ||
| 102 | |I1 to I5|I1 to I5, C3, C4: insertion from both, Edit Macro dialog fields, focus, submit with the default values | ||
| 103 | |E1|E1: screenshots in the editor and in view mode, to compare | ||
| 104 | |E2|E2: one invalid value per parameter type; text parameters are checked by hand | ||
| 105 | |E3, E4, E5, E6|E3 to E6 | ||
| 106 | |||
| 107 | The tools give evidence, not results: read each section against the measure of the criterion and decide. L1, C1 wording, C5 grouping, C6 and C9 stay judgments. | ||
| 108 | |||
| 109 | Record each result in **Edit the audit**. The criteria of the Configuration section have one result per parameter, in the Parameters table of the form. | ||
| 110 | |||
| 111 | === 7. Record the results === | ||
| 112 | |||
| 113 | * Record **OK**, **KO** or **N/A** for every criterion. Leave no criterion empty in a reviewed audit: the results count empty values as Not set. | ||
| 114 | * Do not look for tickets yet: the **KOs and tickets** table at the bottom of the audit lists every KO, and the tickets are linked when you [[report the KOs>>||anchor="HReporttheKOs"]]. | ||
| 115 | * Write the notes as one paragraph per criterion, starting with its ID, like "D4: the macro name gives no result.": the audit shows each one on the row of its criterion. A paragraph that starts with no criterion ID is shown under Other notes. Parameter notes go in the notes of the parameter, in the Parameters table. For each KO, give an example that reproduces it, so that the ticket can be filed from the note alone: for E2, the source of an invalid value the macro accepts silently, like ##{{{ {{warning width="abc"}}Text{{/warning}} }}}##. Write "minor" after the ID of a KO with a small impact. | ||
| 116 | * When a criterion cannot be met because of a technical limitation, record the limitation and the chosen workaround once in the [[Cross-macro findings>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Cross-macro findings.WebHome]], link it from the notes of every affected macro, and mark the criterion as KO. Apply the same workaround to every affected macro. | ||
| 117 | |||
| 118 | === 8. Mark the macro as reviewed === | ||
| 119 | |||
| 120 | Set the status to **Reviewed**, even when every criterion is OK, so that "no issue" never means "not reviewed yet". Fill the **Audit date**: the history of the Usability analysis groups the reviewed macros by this date, and uses the last change of the macro page when it is empty. | ||
| 121 | |||
| 122 | Then **delete the earlier analysis**: the content written on the macro page before the audit, below the audit display. The audit replaces it, so keep only the include line at the top of the page, either by editing the page in the wiki editor or with ##audit_macro.py <macroId> ~-~-delete-earlier-analysis ~-~-apply##. The deleted content stays in the page history. Until it is deleted, the audit shows a warning, and the Usability analysis lists the macro. | ||
| 123 | |||
| 124 | The macro then counts in the progress, the history and the results of the Usability analysis. | ||
| 125 | |||
| 126 | == Report the KOs == | ||
| 127 | |||
| 128 | Once the main batch of macros is reviewed, make sure that a JIRA ticket covers every KO. The [[KOs not reported yet>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.WebHome||anchor="HKOsnotreportedyet"]] table of the Usability analysis lists every KO of the reviewed macros that no ticket covers yet, with the known issues to check: report from it, so that the KOs of several macros can be grouped in one ticket. The **KOs and tickets** table at the bottom of each audit shows the same KOs for one macro, with the tickets that already cover them. A macro can also be reported on its own as soon as it is reviewed, from its audit. | ||
| 129 | |||
| 130 | 1. **Refresh the known issues.** Run ##known_issues.py ~-~-apply##, then sort the issues it lists as To sort in the object editor of the [[Known issues>>doc:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.Known issues.WebHome]] page, so that a ticket filed since the last search is not duplicated. Doing it once at the start of a reporting session is enough. | ||
| 131 | 1. **Link the existing tickets.** For each KO of the table, check the known issues suggested in italics: the ones filed for the macro, alone or with a few other macros, under the same criterion. The issues of every macro are not suggested, since they repeat on every audit. When one covers the KO, use **Add a ticket** on its row: it adds a ticket covering this KO to the audit of the macro and opens its form. Fill the JIRA key, set the origin to Existing, and add the other KOs of the macro the ticket covers. | ||
| 132 | 1. **File the new tickets.** Group the KOs left, see below, and create one JIRA ticket per group. Then record it with **Add a ticket** on the row of one of its KOs, with the origin New, and add the other KOs of the macro it covers. When a ticket covers KOs of several macros, record it once on each of them. | ||
| 133 | 1. **Check the coverage.** A KO leaves the table as soon as a ticket recorded on its macro covers it. The reporting is done when the table says that every KO is covered. | ||
| 134 | |||
| 135 | One ticket can cover several KOs, so there is no need for one ticket per KO. Group the KOs that one change would fix: the same criterion on several parameters, like all the C4 KOs of the macro, several criteria on one parameter, or the same problem in several macros, which the table shows side by side. A ticket about a problem shared by several macros is recorded on the page of each of them. A KO caused by a technical limitation is covered by the ticket of its cross-macro findings entry. | ||
| 136 | |||
| 137 | A KO is written as the criterion ID for a macro criterion, like ##D4##, and as the criterion ID and the parameter ID for a configuration criterion, like ##C4:title## or ##C1:$content##. | ||
| 138 | |||
| 139 | == Tools == | ||
| 140 | |||
| 141 | The tools are Python scripts that automate the setup of an audit and the maintenance of the audit pages. Download [[macro-audit-tools.zip>>attach:Proposal.WYSIWYGusabilityofXWikimacros.Analysis.Audit [email protected]]] and unzip it. Its ##README.md## gives the details. | ||
| 142 | |||
| 143 | * **Requirements:** Python 3.9 or later, and a design.xwiki.org account with edit rights on this proposal. Publishing the pages with ##analysis_page.py## also needs script rights. ##probe_macro.py## also needs [[Playwright>>https://playwright.dev/python/]]: ##pip install playwright## then ##playwright install chromium##. | ||
| 144 | * **Audit instance:** the checks read the local XWiki 18.8.0 at ##XWIKI_AUDIT_URL##, ##http:~/~/localhost:8080/xwiki## by default, as ##XWIKI_AUDIT_USER## with ##XWIKI_AUDIT_PASSWORD##, Admin and admin by default. They write their test pages in its ##MacroAudit## space. | ||
| 145 | * **Credentials:** set the ##XWIKI_USER## and ##XWIKI_PASSWORD## environment variables, or write them as ##KEY=value## lines in a ##~~/.xwiki-credentials## file. | ||
| 146 | * **Known issues:** ##known_issues.py ~-~-apply## runs the JIRA searches again, refreshes the title and status of every known issue, and adds the new ones as To sort. It never changes how an issue was sorted. | ||
| 147 | * **Dry run first:** every script only prints what it would change. Add ##~-~-apply## to write. Every write is a minor revision, so page watchers are not notified. | ||
| 148 | |||
| 149 | The archive is updated each time ##analysis_page.py ~-~-apply## publishes this page, so it always holds the tools that produced the published pages. | ||
| 150 | |||
| 151 | == Maintain the audit == | ||
| 152 | |||
| 153 | * **The criteria change:** the audit keeps using the copy of version 15.17. To move to a new version, run ##sync.py## then ##sync.py ~-~-apply## to update the classes, and change ##CRITERIA_VERSION## in ##analysis_page.py## then run ##analysis_page.py ~-~-apply## to take a new copy. A criterion removed from the criteria page is disabled in the classes, not deleted, so the results already recorded are kept. Audits keep the criteria version they were done against. | ||
| 154 | * **The statistics, the charts or the audit display change:** edit ##MacroAuditStats.xwiki## or ##MacroAuditSheet.xwiki##, then run ##analysis_page.py ~-~-apply##. | ||
| 155 | * **The searches for known issues change:** edit ##QUERIES## in ##known_issues.py##, then run it. The issues found only by a removed search stay on the page. |
