Skip to Content

Office Import Improvements

Last modified by Vincent Massol on 2024/11/19 16:14

 XWiki
 Design
 Dormant
 
 

https://design.xwiki.org/xwiki/bin/view/Proposal/ParagraphNumbering

Description

This document list the requirements and implementation analysis for the office import improvements.

Analysis of Different Options for Office Imports

We evaluated different formats, LibreOffice versions and MS Word HTML as source for the office import. The result is that the currently recommended LibreOffice version 7.1.7 (latest LTS) seems to provide possibilities to realize all requested features while not causing the additional development effort of switching to a new format.

Technical Details of the Analysis

  • LibreOffice 6 (currently distributed in XWiki docker images, see https://jira.xwiki.org/browse/XDOCKER-157)
    • delete Same as below, but heading numbers are not explicitly exported but implicitly as part of ordered lists. Full numbering information would only be possible with an extra filter. As this is extra development work, we won't support LibreOffice 6 for heading numbering.
  • Existing LibreOffice HTML export
    • add Existing implementation well-tested.
    • add/delete We can add filters that are executed within LibreOffice that allow us to extract additional information like heading numbers or paragraph styles. Each of these filters needs to be developed, tested and documented, though.
  • LibreOffice XHTML export
    • add Some more metadata for heading numbers that makes detection reliable without an additional filter (there is a span with class "heading_numbering").
    • add CSS represents actual content styles like font size that could be used for additional detection, e.g., of numbered paragraphs style.
    • add We can add filters that are executed within LibreOffice that allow us to extract additional information
    • add/delete Image import might need adjustments but is probably easy to fix as images are contained inline.
    • delete Will require extensive testing
    • delete Many styles no longer represented as explicit HTML tags like <b> but with some non-semantic class names, thus this will require parsing and interpreting CSS for import and thus significant work for import (currently the CSS is not parsed).
    • delete All lists no longer use actual list styles but contain spans with numbers/symbols. This will require additional work for importing regular lists and guessing for the right parameters for the lists which are basically provided automatically as traditional HTML in LibreOffice HTML export.
  • MS Word HTML
    • add Contains all word styles as HTML classes/CSS styles and thus allows creating an explicit mapping between Word styles and wiki markup
    • add Tabs are represented as spaces for alignment which means that a basic import of alignment tabs works out of the box as long as the font is the same.
    • delete Lists are not represented as lists but sometimes as regular paragraphs that would require custom transformations. Also different variants of lists need to be detected and transformed into XWiki lists. The default mapping doesn't seem obvious.
    • delete Will require extensive testing as the provided HTML is very different.
    • delete Image import is not clear, images are exported as separate files that would need to be uploaded by the user probably unless we automate the MS Word to HTML conversion and capture the images.
    • delete Can't be the default import option for XWiki as it requires MS Word which is non-free and the HTML is not compatible with LibreOffice.

Conclusion: Sticking with the current export format seems to be the best option. We can identify additional metadata using JODConverter filters and thus, e.g., identify paragraph numbers reliably. All other options require significant additional work and could cause a lot of breakage.

Office Import Implementation/Estimation

General

  • Design/architecture - 2d (the time it took to produce the first estimation document that led to the need to make a POC).
  • Note that General Setup, Project Management & Delivery Management is to be added on top and not accounted for in this quotation
  • Automated test and documentation - They are included in the men/days below

Subtotal = 2d

Import Document Name

  • Use the name entered in the import dialog as document title instead of the title from the HTML / some heading content (currently, it is only used for the name of the page)

    CreatePage.png
    The create page screen whose title input shall be used

    • Implementation Just pass the title set to the office import code.
    • Quote: 0.5d

Out of Scope

  • No option to override the title (when advanced user) in order to extract it from the first heading (current behavior before the fix below to use the user-entered title in the create page wizard).

Subtotal = 0.5d

Import UI and Configuration (filters)

Each filter will be an optional filter that includes:

  • Name
  • Short description
  • Possibly longer explanation with a screenshot to demonstrate the effect
  • Test cases with at least one example document that has paragraphs matching the rules and paragraphs not matching the rules. Documents may be shared between tests.
  • Ordering priorities: Each filter define a integer priority (as we currently do for components). If a filter must be executed after another one, it needs to define a larger priority integer

Implementation: Filters may be filters for JODConverter (implementing org.jodconverter.local.filter.Filter) or HTML filters inheriting from AbstractHTMLFilter or filters can also be XDOM transformations. Filters will be implemented with a single java component role (and 3 filter methods), with a single global priority FTM.

  • Load and display the filter sets on the import UI + (for advanced users) possibility to select filters manually

    FilterStyles.png
    Old filter styles UI to replace

    • Implementation: When opening the import UI, a dropdown list of existing filter sets is proposed (the "default" set is listed first and selected). For advanced users, also list all the filters, with the filters from the selected set highlighted. The user then has the choice to unselect filters or select new ones.
    • Quote: 1d
  • Possibility to configure sets of filters that can/must be activated together
    • Implementation: Filter Descriptor Sets + Filter Descriptors as java components + ability to define a Filter Descriptor Set using an XObject with the following xproperties: name + description + comma-separated lost of Filter Descriptor hints.
    • Quote: 3d
  • Load the list of activated filters and execute them in order
    • Implementation: This requires refactoring the internal structure of the office import as currently just a boolean parameter is used for the HTML filtering. In particular, the JODConverter filter chain is configured also just once. The internal structure needs to be refactored to build the filter chain once for every import (this should be possible without significant overhead as the office process is managed in a different object). Further, we need to add the possibility to transform the XDOM result.
    • Quote: 2d
  • Port the existing options to filters
    • Quote: 1d
  • Modify the WYSIWYG Import Office dialog to also display the filters and filter sets

    OfficeImportDialog.png
    The current WYSIWYG office import dialog

    • Quote: 1d

Subtotal = 8d

Implemented filters

General (Implemented in Office Import)

Note that filters depending on contrib extensions (such as the Numbered Reference extension) cannot go in XS.

  • Currently the header/footer is imported once at the beginning/end. This seems to be mostly the header/footer of the first page as in HTML there is just one page. With the filter, header and footer must be removed.

    Footer.png
    Example footer

    • Rules: Page header and footer will be deactivated in the Word document.
    • Implementation analysis: We can turn off the footer and header in LibreOffice for all page styles. Will be implemented as an filter for JODConverter. See PoC filter code
    • Quote:  0.5d
  • Content before the first heading including the table of contents must be removed.

    TOC.png
    Example table of contents that must be removed

    • Rules: Everything before the first heading of level 1 is removed. Content inside tables is not considered here. Option: The first heading to be considered must have "NumberingStyleName" "Outline", which is the style that represents typical heading numbers.
    • Implementation analysis: We can iterate all paragraphs outside tables (but including tables) using a paragraph cursor in LibreOffice. We can query paragraph styles to determine the numbering style. We can then select all text from the start of the document up to the end of the paragraph before the first heading and replace it by an empty string. Will be implemented as a filter for JODConverter.
    • Quote: 1d
  • Currently, every imported paragraph has space between lines. In Word, paragraphs styled with the default style do not have any margin below unless it is explicitly set. Sequence of paragraphs without margin must be grouped into a single paragraph with line breaks.

    Spacing.png
    Example content with different line spacing

    • Rules: Paragraphs that match the following criteria are merged with the next paragraph, separated by simple line break. We match paragraphs that
      • have style "Standard"
      • are not empty
      • have an empty numbering style (i.e., are not a list item)
      • have bottom margin 0
        • Tests show that even though LibreOffice says the bottom margin of the paragraph is 0, the API returns a value of 71 for paragraphs in table cells. It is not clear why this is the case. Using paragraphs with a bottom margin of less than 100 seems to work as a workaround.
      • are not the last paragraph of the document
      • are followed by a paragraph that is of style "Standard", not empty, and has empty numbering style

            the last condition is to ensure that we do not merge paragraphs into following headings, figures, tables or lists.
    • Implementation analysis: This consists of two steps: identifying the affected paragraphs in LibreOffice and merging the affected paragraphs with the following paragraphs in an HTML DOM or XDOM filter.
      • Identifying the paragraphs: We need to iterate both the global paragraphs as well as paragraphs inside table cells. For this, we can iterate over tables, their cells and the cells' paragraphs (see PoC code for iterating over table cells and their paragraphs). We can then select and get the content of the current paragraph and use a second cursor to peek at the next paragraph. Alternatively, we can also simultaneously advance two cursors that are always one paragraph apart. We then need to insert a unique string into each paragraph to identify it. See PoC code for handling a single paragraph
      • In the HTML filter, we would search for all elements containing the unique identifier, remove the unique identifier, and insert its content (children) at the beginning of the next paragraph together with a line break.
      • Note: We need to make sure that this doesn't interfere with the tabs to table transformation.
    • Quote: 2d
  • Convert text defined as Table or Figure captions should be transformed into the numbered references macro. This should include the anchor information from MS Word. Frequently captions do not seem to be marked as such. They should be identified, too.

    Figure.png
    Example figure with caption that is not marked as such.

    • Rules:
      • The first paragraph after an img starts with "Figure" (note: this text depends on the locale, making it configurable might be an option) and continues with a space and a number.
      • For tables the same but with "Table". Tables might also be after the caption.
      • (Optional) Check if the text is also centered.
      • TODO: some proper figure/caption element needs to be analyzed, waiting for input.
    • Implementation analysis: In JODConverter/LibreOffice, we can iterate over all graphics elements or all graphics elements in a selection (like a paragraph) to get all images/all images of a paragraph, see here for some example as macro, this should also work in Java. We can then use a paragraph cursor to go to the next paragraph. We can then check the condition(s) and set some marker and then transform the figure in an XDOM (or HTML DOM) filter. Some step currently seems to filter out anchors of figures. We need to identify the step and see if we can disable it. Otherwise, we need to extract the anchor information before this step is applied. TODO: check how we can identify proper figure environments, waiting for documents to test.
    • Quote: 3d
  • References to figures should be detected and transformed into the numbered references macro.

    FigureRef.png
    Example references to two figures.

    • Rules:
      • If there is an anchor in a caption that is referenced in a link in the text and the link text is a number, the anchor is replaced by a macro that automatically provides the figure number.
    • Implementation: We need to identify all anchors inside figure captions and check where they occur in the text. Wherever they occur, we replace them by a proper figure reference. This works probably best using an XDOM filter.
    • Quote: 2d
  • Alignment tabs are a popular way of formatting text in a table-like format in Office documents, e.g., to format a list of abbreviations and their definitions or a list of areas and a person responsible for each area. Alignment tabs should be replace by an (invisible) table.

    Tabs.png
    Example use of alignment tabs.

    • Rules:
      • A sequence of paragraphs that have empty numbering style and contain a tab character is replaced by a table
      • Each part of the content separated by tab is represented as a table cell
      • (Option) Several tabs without content between them are treated as one tab (Option) but only if a) all lines contain the same number of tabs and b) all lines contain the same sequence of numbers of tabs that are together (a sequence could be "2, 3" if we have "Some content\t\tOther content\t\t\tMore content")
      • (Option) Restrict to paragraphs that match the same requirements as simple line breaks (apart from the requirements for the margin and the next paragraph for the last paragraph of the sequence of paragraphs).
    • Implementation: Identification of tab characters must be done inside LibreOffice as the HTML contains tab characters for indentation that are impossible to distinguish from alignment tabs. We can use a global replacement for this which is already provided by JODConverter. For the last option, we additionally need to mark paragraphs and remove markers again later. That way we could detect blocks that are visually grouped together. Note that again we also need to do this detection in table cells, too.
      • Transformation into a table requires recovering the markers and the range of paragraphs in an HTML filter and replacing it by a table. We should also set a table style.
      • Note: There is a way to create a table in LibreOffice from a selection of text separated by tab characters (as option) in the menu. There is also a method convertToTable in the API but: it expects a sequence (array of arrays) of text ranges to use and an array of cell properties, row properties and table properties. There seems to be very little documentation or examples how to use this method. We might need to select every part between the tab characters to transform them into a table. This might have the same level of complexity as the transformation in the HTML DOM or XDOM.
    • Quote: 4d (Very complex filter due to difficult detection, handling of edge cases like lines with different number of tab characters, creation of the table with splitting of text while preserving formatting of content etc.)
  • (Option) Headings that are marked as "UPPERCASE" text effect are currently imported in whatever case they were typed.

    Uppercase.png
    Example of uppercase heading.

    • Rules: If a heading has style all-uppercase letters in the Word document, transform the characters to uppercase during the import
    • Implementation: This can be checked while iterating over paragraphs by examining the paragraph properties and replacing the string. See (commented out) PoC implementation. Note that this might loose the formatting of the content and possibly also anchors.
    • Quote: 0.5d
  • Mapping of additional list styles or other styles will be implemented (and quoted) on demand.

Subtotal = 13d (with options), 12.5 (without options)

For heading numbering (Implemented in Heading Numbering Application)

Note: The Heading Numbering feature is quoted separately. The cost below is only for the import part.

  • The paragraph/heading numbers defined in the imported office document must be exactly the same after import (including elements without numbers that must stay that way)

    NumberedHeading.png
    Example of a numbered heading.

    • Technical notes While this can easily be done by hardcoding a start value on each element, it is interesting to consider a solution where start values are striped if they match the value that would be computed by default
    • Implementation: This needs both a filter in JODConverter and an HTML DOM or XDOM filter. By default, heading numbers are prepended to each heading. By inserting a string at the start of each heading, we can separate the heading number and the content. The HTML DOM or XDOM filter needs to remove the heading number and the separator from the content while being careful not to remove any formatting that might come after this string. Further, a heading numbering macro might need to be inserted.
    • Quote: 2d (with preserving numbers)
       
  • Manually numbered heading must be imported

    ManualHeading.png
    Example of a heading formatted as default paragraph style.

    • Rules: If a paragraph starts with a number followed by a dot and has a font size larger than the "Standard" style, we consider it a heading and transform it accordingly in the context of numbered headings but not in the context of paragraph numbering.
    • Implementation: This can be implemented as a LibreOffice/JODConverter filter that iterates over all paragraphs, checks the condition, extracts the number and if all conditions match, converts the paragraph into a heading of the appropriate level. It can then format the content the same way as the heading numbering filter expects such that we can use the same filter for HTML. Note: this therefore depends on the heading numbering HTML filter. We need to ensure that we do not mark such headings again using the JODConverter filter of the heading numbering.
    • Quote: 1.5d

Subtotal = 3.5d

For paragraph numbering (Implemented in Paragraph Numbering Application)

Note: The Paragraph Numbering feature is quoted separately. The cost below is only for the import part.

  • The content of numbered paragraphs must be transformed to a numbered paragraphs macro

    NumberedParagraph.png
    Example of a numbered paragraph with an unordered list, a figure and a paragraph without number.

    • Rules:
      • Headings with numbering style "Outline" are transformed into numbered paragraphs
      • Content between numbered paragraphs is put on the same level as the numbered paragraph but without numbering
    • Implementation: The conversion will consist of two steps. In a JODConverter filter, we convert all headings that do not have "NumberingStyleName" "Outline" into regular text body content to simplify the conversion later. Further, we do the same transformation as for heading numbers to recover paragraph numbers (or just run that filter/depend on that filter). See PoC implementation. At the end we need to wrap the whole content into a paragraph numbering macro in an XDOM transformation.
    • Quote: 1d
  • (Option) Automatic detection of the paragraphs/heading style of the current document.

    FollowStyle.png
    Definition of the style "Heading 1" that is followed by "Heading 2" instead of "Text Body" (see first screenshot for heading numbering).

    • Rules: A document is a paragraph numbering document if the paragraph style "Heading 1" is followed by style that starts with "Heading" (in normal documents headings trigger the text body style as next style).
    • Implementation: We can check the condition in a LibreOffice/JODConverter filter, see PoC implementation.
    • Quote: 1d

Subtotal = 1d (without options), 2d (with options)

TOTAL (without options): 27.5 men/days

TOTAL (with options): 29 men/days

Useful Resources for Open/Libre Office Programming/for JODConverter filters

Attention: this kind of filtering is not supported in the online converter, see https://github.com/sbraconnier/jodconverter/issues/118

Where the filters are applied in XWiki: https://github.com/xwiki/xwiki-platform/blob/7b552f38974ef8ccbdd57f29b06a9049bb600376/xwiki-platform-core/xwiki-platform-office/xwiki-platform-office-importer/src/main/java/org/xwiki/officeimporter/internal/server/DefaultOfficeServer.java#L153

The following resources were collected while developing prototypes for the above-mentioned filters and could also help with the final implementation:

Analysis of Provided Documents

The following details the analysis of three provided documents that lead to the proposal of the new filters. It is summarized also in the section which import format to choose. LO6/7 means LibreOffice 6/7 in the following. The "existing import" in this text is based on LibreOffice 6 and thus doesn't reflect the recommended version. The analysis of the existing import has been partially repeated with LibreOffice 7 where differences existed to confirm that the import works as expected from the differences in the HTML produced (which is reported below), the results just haven't been updated in the list below.

  • Document 1
    • numbered headings - but the heading are actually bold paragraphs with increased font size (but don't have any heading style)
      • Existing import:
        • Numbers and bold text are preserved but not the font size
      • MS Word HTML:
        • p/b/span-tags without any semantic styles or tags that could be interpreted
      • LO6 + 7 HTML: p/font/b-tags with numbers in the text
      • LO6 + 7 XHTML: p with a class that has bold and font-size CSS associated
    • vertical lines and numbers on the side (e.g., page 7) are imported as images
      • existing import: imported as images without alignment
      • MS Word: present as tables with images
      • LO6 + 7 HTML: present as images in the middle of the text
      • LO6 + 7 XHTML: only numbers in strangely positioned div + p tags in the middle of the text
    • Lists in 7.1 with a) and -, lists in 8.1 with dots
      • Existing import:
        • Lists in 7.1 just as text, lists in 8.1 recognized as lists
      • MS Word HTML:
        • Lists in 7.1 seem to be regular paragraphs, lists in 8.1 are paragraphs with some list-style attributes (most-lists: l3 level1)
      • LO6 + 7 HTML: Lists in 7.1 are regular paragraphs with some font tags for styling, lists in 8.1 are ul/li elements
      • LO6 + 7 XHTML: Lists in 7.1 are regular p-tags with classes with some CSS styles associated, Lists in 8.1 are ul/li elements with custom list characters in an extra span that seems hard to identify (has some class but its name seems rather random)
    • Aligned text in 4 and 8.1
      • Existing import:
        • Alignment lost, just a single space
      • MS Word HTML:
        • Alignment preserved using spaces and a style-attribute indicating the number of spaces. Tab stops indicated in pt at some parent tag.
      • LO6 + 7 HTML: Tabs are represented as tab character
      • LO6 + 7 XHTML: present as a fixed number of spaces without any further markup.
    • Conclusion: existing import already works nicely apart from the vertical lines and numbers and the tab-aligned text, will need heuristics to identify headings and their numbering based on styles.
  • Document 2
    • Table of Contents
      • Existing import: Just some paragraphs with links
      • MS Word: p class=MsoToc1/2/3
      • LO 6 + 7 HTML: p/b inside div id="Table of Contents1/2/3" (for figures/tables)
      • LO 6 + 7 XHTML: just some paragraphs with P15/16/17
    • numbered paragraphs
      • Existing import: Imported as nested ordered lists with headings inside, numbers start from 1 if there is any content outside numbered paragraphs
      • MS Word HTML: h1/h2-tags with mso-list:l14 level1/2/3 lfo22, numbers in extra span with most-list:Ignore
      • LO6 HTML: ol with h1/h2 inside with start=... to indicate start number
      • LO6 XHTML: ol/li/h with a class for list-style: none, numbers in a span without a special class but inline styles (float:left, display:block, min-width...)
      • LO7 HTML: just h1/h2 with numbers inline without any spacing
      • LO7 XHTML: just h1/h2, numbers inside in a span with class="heading_numbering"
      • Word + LO XHTML: styling uses correct (small) font size for headings
    • paragraphs without number in section 4
      • Existing import: Imported as regular paragraph
      • MS Word HTML: p with BodyTextDefinition class
      • LO6 + 7 HTML: p with class body-text-definition
      • LO6 + 7 XHTML: p with some class P13
    • list in 5.1.1
      • Existing import: Imported as regular list (outside the numbered paragraph)
      • MS Word HTML: p with mso-list:l6 level1, span with style most-list:Ignore and list symbol as content
      • LO6 + 7 HTML: ul/li/p tags outside ol
      • LO6 + 7 XHTML: ul/li/p with list-style: none and custom list marker symbols in a span without class but inline styles
    • Figure 1 caption:
      • Existing import: heading with some ids
      • MS Word HTML: p class=FIGURE, no special markup around figure number
      • LO6 + 7  HTML: h1 class=figure
      • LO6 XHTML: p class=FIGURE
      • LO7 XHTML: p class=P19
      • Note that figures seem to have ids "FX" and there are links in the text that use these IDs!
  • Document 3
    • numbered headings
      • Existing import: headings inside ordered lists, numbering not preserved
      • MS Word HTML: h1/2 with mso-list:l11 level1/2, number in span with ms-list:Ignore
      • LO6 HTML: ol/li with start=X for numbers, but numbers seem to be incomplete, only start=x attributes for innermost level (e.g., a h2 has an outer <ol> without number and an inner <ol> with start=2 but the number should be 3.2)
      • LO7 HTML: just h1/h2 with numbers inline
      • LO6 XHTML: ol/li/h with list-style=none, number in span with inline styles but no class
      • LO7 XHTML: just h1/h2 with numbers in span class="heading_numbering"
    • TOC
      • Existing import: text with some bold/italics markup, no links
      • MS Word HTML: p class="MsoToc1/2/3"
      • LO6 + 7 HTML: div id=Table of Contents 1
      • LO6 + 7 XHTML: table, numbers missing apart from level 3!
    • lists
      • Existing import: regular lists
      • MS Word HTML: p with class mso-list: l16 level1 lfo26, note that there is no surrounding container though
      • LO6 + 7 HTML: ul/li/p
      • LO6 + 7 XHTML: ul/li with list symbol in span with inline styles, no class
    • Figure 2
      • Existing import: figure present, caption below figure without markup
      • MS Word HTML: seems to contain some code as shap as vml (see https://docs.microsoft.com/en-us/windows/win32/vml/msdn-online-vml-introduction), gif from directory (not present), caption in separate paragraph, a current Office 365 also doesn't display the VML but a small pixelated figure.
      • LO6 + 7 HTML: external gif image in p, caption in separate p
      • LO6 + 7 XHTML: image in data:image/png-attribute, caption in p class=P31/P43

 

About

About

Support

Platform

User Guide

Admin Guide

Developer Guide

Projects

XWiki

Extensions

Other

Contribute

Status

Practices

Under the Hood

Get Involved

Get Connected