Description

Some work has been done here: https://github.com/fmancinelli/xwikifs 

Objective

Directory structure for XWiki XML pages, that:

- separates script and attachments files from the XML
- names the new external files with the proper extension

Some requirements are:

- it should be possible to build the final XAR with Maven from this structure
- the structure should be (as) easy (as possible) to navigate in an IDE style view
- it should be easy to add a new script or attachment on an existing structure
- it should be easy for GitHubApp/SVNApp to check of the content of a wiki is in sync with the source structure in GitHub/SVN with a GitHub/ASVN api
- it should be easy for a sync tool to compare the source structure with a content of a wiki with the REST API

Possible solutions

Caleb Method

Some work has been done here: https://github.com/xwiki-contrib/node-xwikimodel and some example here https://github.com/xwiki-labs/xwiki-tools-example

Version One

Version one contained no directory structure at all, it was nothing more than an API for scripting extensions. Here you can see an example usage of version one API.

Version Two

Version Two is built on top of Version One to make a canonical representation of an xar extension in a directory tree, specifically it aims to be reversible in that files in a wiki can be dumped into this format.

All data representation is in Javascript (not JSON). Though the content is often "data-like", the data is intentionally defined in Turing Complete language so as not to restrain those applications which have valid needs to do additional processing when creating the content. In each directory there may be a this.js file, the file is executed with the relevant object and is able to set properties of that object. If additional files exist in that directory, fields in the object will be set to the content of those files. For example: /MySpace/MyPage/this.js might contain doc.setTitle("my document") which would set the title of the document but if /MySpace/MyPage/content.xwiki20 exists, it is as if doc.setContent() was called with whatever is contained therein. The filename extension is ignored so that one can use whatever extension makes sense for the content type.

In the root of the directory tree, this.js defines the Package. The Package definition allows setting of the package name, description and extensionId as well as the name of the final XAR file which will be created. See: example of package definition.

Along side the package definition, there are directories representing the spaces in the XAR, inside of each space are another set of directories representing the pages in that space. Each page directory may optionally contain a this.js file which is used to set the fields in the document. See a basic example of a document definition. There are two special types of subdirectory under a document directory, one is attachments and the other is objects.

The attachments subdirectory is simple, each file contained therein is converted into an attachment to the containing document. The objects subdirectory on the other hand is less so. The objects directory contains a subdirectory for each object, the name of the subdirectory is the name of the object XClass followed by an underscore and the object number. Inside of each object directory is a this.js definition which sets fields on the object and optionally additional files which are used to populate fields in the object. For example, in this JavaScriptExtension object, the Object Definition populates three out of four fields in the object while the code field is populated from a separate file called code.js.

In the event that this directory structure proves inadequate for a particularly complex extension, the extension can make use of the Javascript API to "compile" the package, of course this is not advisable since it is irreversible. Here you can see an example XWiki document where a particular attachment is constructed programmatically by concatenating the content of all files in a directory. Note that as with any other Document Definition, XWikiDoc() function is called by the javascript but in this case it is not called until after all asynchronous processing is complete.

Advantages:

• Ease of comprehension, simple to understand what a file means by looking at it.
• Covers most common cases without resorting to advanced scripting of the extension creation process.
• Flexibility in the event that an extension cannot be reasonably represented without resorting to code.
• Ability to evolve new directory structures (Version Two is just a single 283 line script which uses Version One as the back-end)
• Fast, builds a XAR 27 times the speed of Maven.

Disadvantages:

• There is no automated way to create this structure from code in the Wiki (TODO)
• It is more difficult for the GitHubApp/SVNApp to check the repository as it would need to understand the JS script.
• Easy to accidentally begin relying on NodeJS-specific API in your extension definitions. (xar linter?)
• Still lacking a Maven job for building extensions (TODO)

Challenges:

• XClass definitions are traditionally shipped with XObjects in the xar, currently there are a handful of pre-defined XClasses included in xwiki-tools but it is not currently possible to create XObjects of additional XClasses. We need a coherent way to write XObjects without duplicating the XClass definitions everywhere.

• Dumping a XAR or Wiki to a file structure necessitates a certain amount of guesswork to make a file structure which is human readable. For example the content of a large syntax 2.0 document should probably be in a file called content.xwiki20 but the title is probably small enough that it should be an inline doc.setTitle() call in the Document Definition. JS Extension objects have a field called code which should be represented as a file called code.js while the same code field in a CSS Extension object should be represented as code.css.

Automatic Flat Structure for XML

Explode all XML files into external files for content fields, textarea fields and attachements in a flat structure or inside a directory per page.

Advantage:

• Fully automatic
• Can work with GitHubApp/SVNApp although it's more complex than today
• Automated way to migrate current project structure
• XAR format can check format
  
  Disadvantage:
• XML and code files in the same directory
• For some code files you cannot detect the type automatically (would need an object)
• File naming is not user controled (you cannot choose your fields names)

Manual Flat Structure for XML

User decides which fields to extract as individual files using an special syntax in the XML

Advantage:

• User can decide the file names which is good
• No real standard for the structure
• XAR format can verify if structure is valid but cannot really create the structure
  
  Disadvantage:
• Not Fully automatic
• XML and code files in the same directory
• No Automated way to migrate current project structure (proposal is possible)
• More complex for GitHubApp/SVNApp because you don't know which fields should be extracted and you need to look at which have already been extracted

Structure for XML with mapping file

User decides which fields to extract as individual files using a mapping file 

Advantage:

• GitHubApp can handle it
• User can decide the file names which is good
• The directory structure of the code can be fully decided by the developer
  
  Disadvantage:
• Not Fully automatic
• GitHubApp cannot decide the structure itself, it needs human help
• Need best practices
• No Automated way to migrate current project structure (proposal is possible)

 Code Structure
 --
\\ src/main/mapping/mapping.txt
 src/main/resources/ProjectCode/JSExtension.xml
 src/main/resources/ProjectCode/Transations.xml
 src/main/resources/ProjectCode/CSSExtension.xml
 src/main/resources/ProjectCode/Groovy.xml
 src/main/resources/ProjectCode/ProjectClass.xml
 src/main/resources/ProjectCode/ProjectSheet.xml
 src/main/resources/ProjectCode/ProjectTemplate.xml
 src/main/xwiki/project/translations.properties
 src/main/xwiki/project/jsextension.js
 src/main/xwiki/project/cssextension.css
 src/main/xwiki/project/projectsheet.xwiki
 src/main/xwiki/project/groovy.groovy
 
\\ Mapping file
 
 ProjectCode=project
 ProjectCode.JSExtension:XWiki.JSExtensionClass:code=jsextension.js
 ProjectCode.CSSExtension:XWiki.StyleSheetExtensionClass:code=cssextension.css
 ProjectCode.Translations=translations.properties
 ProjectCode.ProjectSheet=projectsheet.xwiki
 ProjectCode.Groovy=groovy.groovy--


 src/main/mapping/mapping.txt
 src/main/resources/ProjectCode/JSExtension.xml
 src/main/resources/ProjectCode/Transations.xml
 src/main/resources/ProjectCode/CSSExtension.xml
 src/main/resources/ProjectCode/Groovy.xml
 src/main/resources/ProjectCode/ProjectClass.xml
 src/main/resources/ProjectCode/ProjectSheet.xml
 src/main/resources/ProjectCode/ProjectTemplate.xml
 src/main/xwiki/project/l10n/translations.properties
 src/main/xwiki/project/js/extension.js
 src/main/xwiki/project/css/extension.css
 src/main/xwiki/project/sheets/project.xwiki
 
 Mapping file
 
 ProjectCode=project
 ProjectCode.JSExtension:XWiki.JSExtensionClass:code=jsextension.js
 ProjectCode.CSSExtension:XWiki.StyleSheetExtensionClass:code=cssextension.css
 ProjectCode.Translations=translations.properties
 ProjectCode.ProjectSheet=projectsheet.xwiki
 ProjectCode.Groovy=groovy.groovy