Description

• Background
  • The problem

Background

This design proposal tries to define an extension mechanism for inserting content into the interface, thus allowing for Interface Components to be defined, and for Applications to add content in the existing interface.

The problem

Last year (2007), XWiki evolved from a monolithic project, where all the java code, the skin files and all the wiki documents are in one place, to a module-based collection of projects. On top of the platform, clearly separated into the core and web components, we have several plugins and applications, which can be combined to form bigger projects, like XWiki Enterprise or XWiki Watch. Still, a project contains a lot of wiki pages that cannot be easily reused. For example, the XE documents can be separated into several applications, like the Blog application, the Presentation application or the Administration application, but these applications cannot be completely extracted into distinct build modules. And in order for a product to add something to the main page or change the preferences, it must completely override the base documents, usually duplicating the old content and adding only little new content.

While the Presentation applications could be completely separated, without any traces in the skin or in the other applications, most other applications have strong bonds with the platform. For example, the blog is connected to the Navigation application (the Dashboard on the main page lists the most recent articles), with the existing skin files ( elements pointing to the generated RSS), it adds new skin templates (rdf.vm) and new css rules (for the article footer)...

One easily observes the need for providing some extension mechanism for extending various User Interface related components: the velocity templates, the css rules, the javascript, and, most important, the existing interface, generated either by the velocity templates of the skin, the wiki rendering engine, or by the wiki pages belonging to another application.

The bad alternatives

For a programmer, the first idea is to have something similar to the service interface <> service provider paradigm. List some extension points, like "the list of css files", "the list of javascript files", "the menu entries", "the document extra area", "the list of administration tabs", etc, and an application can register parts of it as items that can go in one of the above lists. The problem with this approach is that we can never foresee all the possible aspects an application writer will want to extend.

For example, when Radu implemented the Google Docs Integration plugin, he was confronted with the following problem: how to add a button to open a table into Google Docs? There is no way to intercept the Radeox rendering process, so that the XWikiTable macro can add that button, other than overriding the TableMacro class, and nobody ever thought that one might need sometime to put something else before the generated table. Fortunately for him, plugins can alter the generated output before sending it to the client, so a simple tokenizer can detect the generated table elements and insert some extra HTML.

This solution does not always work, though, as an extension might need the velocity/groovy/radeox context at a certain moment in the rendering process. In the previous example, the plugin can wrongly detect other tables written by the user as plain HTML, and will generate buttons in the wrong place, breaking the real table editing, too.

As another example, imagine that the blog application didn't print the permalink, and somebody wanted to make a very small application that does exactly that: insert a permalink in the article footer. Given the fact that an article is part of an application, and not a platform feature, that definitely isn't one of the platform extension points, and post-processing the generated HTML doesn't help, as it would be more costly to determine to which article does a piece of HTML code belong to.

Even if we could foresee all the possible extension points, we would have a lot of calls for listing and inserting extensions, most of these calls being unneeded.

The good alternative

Having some experience with Firefox extensions development, I really liked the way extensions work there. Basically, an extension can insert a piece of code everywhere. Well, almost everywhere, meaning that as long as something has an ID, it can be overlayed. An interface extension specifies what does it want to extend, by copying the tag and the id of that element, what it wants to add, as XML content, and where it wants to add that, as an insertbefore/insertafter hint. On top of that, an extension can provide a list of Javascript files that are active in the browser, a list of stylesheets that are applied to the whole application, a list of static files that can be used in the interface/css, and a list of translation documents for internationalizing the extension.

I would like to have something similar in XWiki, even more advanced. An Interface Extension:

• can specify what should be extended, not just as an ID, but as a CSS or XPath selector (in the beginning, only a simple subset of CSS selectors will be used, and later we can extend the set of supported selectors)
  • also giving hints, for example before what other extension, immediately after or before the targeted element, or inside before the closing tag
• contains some content that must be inserted
• Things that increase the complexity, but would be nice to have: needed rights (can be done with scripted rights check inside the content), enabled for actions (can be part of the selector or in a scripted check), enabled for space(s), enabled for documents with a type of object attached...

Interface Extensions (IXs)

IXs are stored as standard XObjects. Usually, an Application will have one or more XClasses, with their templates and sheets, some navigational documents with scripts processing the structured documents, an internationalization bundle document, some interface extensions and some skin extensions. When importing such an application, the documents containing IX objects are also imported, and the extensions inside them are recognized as extensions and are further used in the interface. A document can have more than one IX inside; in this case, each object is considered a distinct extension.

Extensions are inserted dynamically, while parsing the content. This is needed because the renderer's context must be captured when inserting an extension.

IX class:

• name: string
• insertionPoint: string, selector
• position: staticList: before, start, end, after
• hint: optional string, should be something like ("before(token)" | "after(token)")*
• content: textArea, contains the extension code
• cache: staticList: never, perPage, global
  Each IX has a name, which is used for displaying the extension in an interface manager. It is needed because more than one IX can be in a document, so the document name is not enough for distinguishing an extension.
  The selector should be simple at first, a subset of the css selectors, which will later be expanded as needed, keeping an eye on performance. So at first it should be one of {element, #id, .class}. The selector will be matched against the rendered content, and will trigger the extension insertion. Let's call the current matching element the trigger.
  The position specifies where, relative to the trigger, should the IX be inserted. before means before the start tag of the trigger, start means inside the element, right after the start tag, end means inside the element, right before the end tag, and after means after the closing tag of the trigger. The default should be start.
  Given that more than one IX can specify the same selector and position, there should be a way to sort the different extensions that will be inserted in a trigger. Thus, an IX can specify a relative ordering with other extensions, using the hint property. When an ordering hint cannot be resolved (because the extension is not installed), then that hint should be ignored. When no order requirements are specified between two extensions, the default ordering should be alphabetical. This artificial ordering is needed so that the results are deterministic and predictable.
  After an IX is triggered, and the exact position is determined (after evaluating the position and hints), the extension content should be parsed, rendered and inserted in the already rendered content. For the moment, the extension should be rendered by all parsers/renderers executed up to the currently active renderer. This allows an extension triggered during the radeox processing to also include velocity content, and prevents double radeox rendering for an extension triggered during velocity processing.
  In some cases an extension might only add some static content (for example a download link, a copyright notice, or a google analytics snippet), and because the rendering process is costly, the IX resulting output could be cached. Thus globally static content can be placed in a global cache, which is refreshed only on time periods (for a copyright notice, or a logo, for example), can be cached on a perPage basis (for an export link added to the menu, for example), or can never be cached, as it always depends on the context (for adding a "number of comments" in the article footer, for example). For IXs included in the document content, the document cache policy has priority over all extensions, meaning that a document with a setCacheDuration(duration) statement inside will be cached for that duration regardless of any extensions that should not be cached.
  Given that extensions can also be extended, triggering an extension with a lower cache tolerance inside an extension with a higher cache tolerance should set the lower value for the outer cache, too. For example, including a "perPage" cached extension inside an "global" cached extension should cause the outer extension to have an implicit "perPage" cache policy.
  The default value for the cache field should be "never".
  Access rights must also be taken into account, so that an IX cannot display data that the user would not otherwise have access to, or that programming rights don't leak from or to the interface extension. The programming rights should be checked on the IX document, thus:

• IXs should be executed in the default security model (programming rights required for privileged calls, access only through the APIs);
• displaying an IX without PR (defined in a document without programming rights) in a document with PR should not execute restricted code;
• displaying an IX with PR in a document without PR should execute the restricted code.

Implementation details

Main idea: Renderer => SAX stream => IX controller[SAX event handler] {check for matching selectors, evaluate extension order, render => recursive}

Currently, we are executing the renderers in order, passing strings between them. An exception is when using $xwiki.render, which causes all the renderers to be called for generating a piece of text which is inserted inside an already running renderer output. Each renderer is fully executed without interruptions, only the final result being of interest.

In order to achieve the live IX insertion, capturing the current context, the rendering should be changed. A renderer's output is transformed into a SAX input source, which is then redirected to the IX controller (IXC). The renderer should be forced to flush the output buffer whenever the context changes, and wait for the IXC to process all the newly generated content. This is not possible for all renderers, but for velocity, the main rendering engine that has a dynamic context, it can be done.

The IXC is notified whenever there are start and end tags detected in the output, and checks the list of IX selectors against such a tag. If there are one or more IXs triggered, the order in which they should be applied is evaluated, then each of these extensions is rendered using the current context. The process is recursive, but would be limited to a reasonable amount of recursive calls, as normally one could extend the default interface, and maybe one or two levels of extension extension.

The process should be optimized using caches. Thus, the list of extensions should be cached, the list of IX selectors should be cached, the extension ordering should be cached (the order is global and always the same), the IX rendered content should be cached. These caches should be invalidated by external events, using the notification mechanism, and not by polling for changes.

Afterwards

New skin model

The skin should be completely refactored, as to include as few elements as possible, and the "paint" should be applied as generally as possible. The rest of the interface will be generated by a special kind of IX, called Interface Components. The different between application interface extensions and interface components is that the former sustains a larger collection of documents that make up an application, by helping the application get integrated in the UI, while the latter is not part of an application, but is a standalone extension whose main purpose is to add something to the rendered document.

A basic skins should ideally contain (as markup) just a simple XHTML skeleton, with a somewhat filled in head and an empty body declaration. We could then have extensions for the different layouts (h+3c+f, h+c+f, gh+sh+2c+f, ...). Another extension could specify the fact that we have an action menu, or a navigation menu. The panels application could be completely separated, as the panels will be inserted by an extension triggered by a "leftcolumn" or "rightcolumn" selector, and not by the view.vm core template.

Generally, each component currently seen in the interface should be a pluggable extension. Using a simple administration/configuration interface, the admin could choose what to be included in the interface. A different footer? Remove the "Default footer" IX which shows the creator, the XWiki version and the copyright notice, and install the "Copyright footer" which shows only the copyright notice. Add a navigation menu which can easily be defined using an AJAX drag'n'drop interface? Just install the "DnD navigation menu" extension. Want to disable the attachment list/upload form at the end of the document? Just disable the "Attachments" extension.

In order for an extension to be reused across more than one skin, first the base skin should define the "paint" (look and feel) using an "open world" paradigm, and not a "closed world" one, thus adding something new in the interface will automatically pickup the right colors, without needing new styles. If that is not enough, then an IX can also contain one or more skin extensions (SX, to be detailed in another proposal). This allows a skin to define just the generic look and feel, and not the way each component is styled. Of course, a skin will have to make some assumptions regarding how the interface will look, and contain styles for stuff that won't always be present in the interface, but might be added by an interface component.

For even easier style reuse, we could define some velocity util/variables that hold common colors, like $skin.text, $skin.background, $skin.title, $skin.link, $skin.highlightText, $skin.contrastBackground...

Development practices

When enabling the platform for interface extensions, we have to make sure that extensions are compatible with as many releases as possible, and with as many skins as possible. To achieve this goal, we need to carefully design the interface structure, choosing class names and ids that should be semantically valid in any skin (like "header", "logo", "breadcrumbs"), and not just a something needed for styling one skin (like "b2", "redPixel", "tlDiv"), documenting these classes/ids (class name: appears where, what does it mean and what it should hold, in what version was it introduced, and if it is planned for removal in what future version). Thus, an external developer who wants to extend something can look if the place he wants to extend is documented, thus stable, if he could use that element as an extension point, or if he just found something unstable and he should search for a better insertion point.

This should be done in two phases. The first one is designing the empty skin and the ids/classes it should contain, without looking at the current skin. The second phase is porting the current skin(s) to the new mechanism, and changing/adding ids and classes as we detect the need for them.

When designing the interface, we should take into account: clean and semantic markup, usability, accessibility, SEO, look and feel.

Refactoring

After the mechanism is in place, we need to refactor the skin according to the development practices, defining the core interface structure, extracting interface components into different IX applications, write the skin administration interface, and finally extract the XE applications into distinct build modules.

With IX and SX, an application could be completely separatable into a build module, which can be packaged as a XAR, which, when installed in wiki, would contain all the needed documents for adding new javascript and css files, and changing the interface as needed, without affecting the platform skin files.