RealtimeLivetable

Last modified by Vincent Massol on 2024/02/26 17:54

 XWiki
 Design
 Active

Description

This page lists the specifications of the new live table.

Functional Requirements

R1: Load

  • Load the data asynchronously (don't block the page load)
  • Show a loading animation

R2: Display

  • Display the data as a table (with visible table header, rows and columns)
  • Responsive display on small screens
  • Be able to mark columns that can be hidden when there's not enough horizontal space
  • Display the data as a grid (card layout)

R3: Pagination

  • Load only a limited set of rows (one page)
  • Show the total number rows
  • Be able to change the number of rows displayed per page (10, 15, etc.)
  • Show page numbers vs. show only First / Next / Previous / Last links

R4: Sort

  • Be able to sort the live table rows by a single column, both ascending and descending
  • Be able to sort on multiple columns
  • Indicate the sort column and the sort order (ascending / descending)

R5: Filter

  • Be able to filter the live table rows by one or multiple columns
  • Indicate the columns with active filters
  • Have a single filter/search input vs. have a filter per column
  • When filtering by multiple columns, use the AND operator (match all)
  • Be able to specify multiple constraints for a single column, and allow to choose whether to apply all (AND) or any (OR)
  • Be able to filter by exact value, partial value, prefix, suffix, less than, greater than

R6: Selection

  • Be able to select live table entries one by one, even from different pages
  • Be able to select all / none of the entries from the current page (loaded entries)
  • Be able to select all / none of the entire entries (including entries that haven't been loaded yet)

R7: Actions

  • Be able to perform actions that target a single row (e.g. view details, delete a row)
  • Be able to perform batch actions that affect the current selection of rows
  • Standard actions: view, edit, delete, copy, export

R8: Bookmark

  • Be able to bookmark or share the current state of the live table (pagination, sort, filters, selection) with other users (through the URL)
  • Be able to obtain the wiki syntax needed to recreate the current state of the live table (to copy & paste in another wiki page)

R9: Synchronization

  • Be able to see in real-time the changes done by other users through the same live table (e.g. if another user deletes a row then update my live table to remove that row)

R10: In-place edit

  • Be able to edit the live table data in-place
  • Be able to add a new live table entry from the live table

Architecture

The following components are involved:

  • A Velocity macro that loads the required JavaScript modules and CSS stylesheets and outputs the HTML placeholder
  • A wiki syntax macro that calls the Velocity macro
  • A generic/reusable data source that retrieves the data from wiki pages and their objects, and that can be extended or customized
  • A (responsive) stylesheet
  • Multiple JavaScript modules:
    • Core: looks for the HTML placeholder in the DOM and enhances it, then fetches the data and fills the table
    • Filters: separate modules that enhance the live table filters (e.g. help the user pick a date range to filter a date column)
    • Displayers: separate modules that handle the display of the data inside the live table (e.g. user displayer, date displayer, etc.)
    • Synchronization: the module that handles the live table synchronization (real-time updates)

Would be nice to also have a wizard to help the users generate the live table configuration.

Let's see how the listed components interact with each other:

  • The user inserts the live table wiki syntax macro in a page.
  • When the page is rendered the wiki syntax macro is executed (on the server).
  • The wiki syntax macro then
    • reads the live table configuration (from its macro parameters, or from its macro content or from another source)
    • and calls the live table Velocity macro, passing the configuration.
  • The Velocity macro (on the server)
    • includes the required JavaScript modules (core, filters, displayers, synchronization), depending on the live table configuration,
    • and the stylesheets,
    • then outputs the HTML placeholder, also passing the live table configuration for the client side using HTML data attributes
  • The browser parses the page HTML and loads the JavaScript and CSS resources
  • When the live table core JavaScript module is loaded
    • it looks for the HTML placeholder in the DOM and reads the live table configuration
    • enhances the HTML structure
    • makes and HTTP request to fetch the live table data (using the data source specified in the live table configuration)
      • (on the server, the data source receives the HTTP request and executes the necessary database queries to gather the data)
    • fires the events needed to trigger the initialization of the live table filters (which are handled by separate JavaScript modules)
    • fires the events needed to trigger the display of the live table data (based on the configuration of each column)
    • initializes the real-time session (synchronization)

TODO: Would be nice to have a diagram to view the interaction.

Let's look at each component in detail.

Wiki Syntax Macro

{{livetable ... /}}

The wiki syntax macro is meant to be used by both simple users and advanced users so it needs to have a good balance between easy to understand parameters and more advanced parameters that have good default values. The live table settings could be read from:

  • macro parameters and/or the macro content (= in-line configuration source)
  • an external configuration source (e.g. from some dedicated objects attached to a wiki page)
  • both, using the external configuration source as a fallback

Here's a list of common live table settings that the user should be able to specify when calling the wiki syntax macro:

  • the data source, can be specified as:
    • the reference of a wiki page that holds a class definition (the default / generic data source is used)
    • the reference of a wiki page that implements a custom data source (could extend the default data source)
    • a URL for a custom data source
  • the list of columns
  • the columns to sort the live table on initially, including the sort order (ascending / descending)
  • the maximum number of rows to display per page
  • the translation key prefix
  • filter values
    • the values used to pre-fill the live table (visible) filters
    • and the values passed to the data source in order to restrict the data set (hidden filters)

Based on this the wiki syntax macro could have the following parameters:

{{livetable

  ## Optional parameters that map to the corresponding HTML attributes, helping developers customize the live table
  ## styles and behavior.
  id="foo"
  class="bar"

  ## Optional parameter that specifies the source of the live table data. The parameter value is a component hint. A
  ## default implementation is provided (mapped to the current XWiki.LiveTableResults page).
  source="default"

  ## Optional source parameters, used mainly to apply hidden filters (i.e. filters that cannot be removed from the UI).
  sourceParameters="className=XWiki.XWikiUsers&location=Users"

  ## Optional comma-separated list of columns. The default list of columns is specified by the data source.
  columns="one,two,three"

  ## Optional comma-separated list of columns to sort on. Sort order can be specified.
  sort="one,two:desc"

  ## Optionally initialize the visible live table filters (the users can change these values from the UI).
  filter="one=blue&three=pending"

  ## Optionally limit the number of live table rows displayed.
  limit="10"

  ## Optional translation key prefix. Should be used only to overwrite the default labels.
  translationPrefix="foo.livetable."

/}}

Check this page for a related proposal. Note that column properties (e.g. whether the column is filterable or sortable) are not included in the list of macro parameters because they should be specified by the data source.

The wiki syntax macro can be used directly in wiki syntax (e.g. from the Wiki editor) but it can also be inserted from the WYSIWYG editor, using the Insert Macro wizard. We need to make sure the macro parameters are easy to set by simple users and for this we need to provide pickers (e.g. a page / xclass picker for the data source).

The wiki syntax macro serves as a wrapper (facade) for the Velocity macro, so it doesn't know and it doesn't care how the live table is implemented. See the Documents Macro for an example of such a wiki syntax macro (the code can be found in the XWiki.DocumentsMacro wiki page).

Velocity Macro

#livetable('foo' $columns $columnsProperties $options)

This is meant to be used only by developers. All the needed information should be passed through macro parameters. The goal of this macro is to include the required JavaScript and CSS resources and to output the live table HTML placeholder, based on the provided live table configuration. It must not depend on wiki pages, because we must be able to use this macro on an empty wiki, from a Velocity template.

The Velocity macro is aware of the JavaScript widget used to implement the live table and the HTML output it generates might have to be specific to this widget (to meet its expectations). So changing the widget used to implement the live table will require updating the Velocity macro (but should not require an update of the wiki syntax macro or the data source). The Velocity macro is independent of the data source specified in the live table configuration.

See the existing live table Velocity macro for more details.

Data Source

We have to provide a default data source that retrieves data from wiki pages and the objects attached to them. This default data source is generic and can be parameterized (the users can pass configuration parameters through the live table configuration). Developers can extend the default data source or write their custom data sources. The data returned by the data source should include:

  • the number of rows available on the server (needed for pagination)
  • the offset and limit used (for pagination)
  • a list of rows (the data to display on each table row)
    • something to identify the row (we need this for synchronization most probably)
    • access rights (whether the current user can edit, delete, etc. this row)
    • URLs of supported actions (edit, delete, copy, etc.)
    • the list of cells (the data to display on each cell)
      • additional information that may be needed to display the table cell (e.g. data type, pretty name, etc.)

The JSON currently looks like this:

{
 "totalrows": 35,
 "returnedrows": 15,
 "offset": 1,
 "rows": [
    {
     //
     // Identify the row
     //
     "doc_wiki": "xwiki",
     "doc_fullName": "Blog.BlogIntroduction",

     //
     // Actions & access rights
     //
     "doc_viewable": true,
     "doc_url": "/xwiki/bin/view/Blog/BlogIntroduction",

     "doc_hasadmin": true,

     "doc_hasedit": true,
     "doc_edit_url": "/xwiki/bin/edit/Blog/BlogIntroduction",

     "doc_hasdelete": true,
     "doc_delete_url": "/xwiki/bin/delete/Blog/BlogIntroduction",

     "doc_hascopy": true,
     "doc_copy_url": "/xwiki/bin/view/Blog/BlogIntroduction?xpage=copy",

     "doc_hasrename": true,
     "doc_rename_url": "/xwiki/bin/view/Blog/BlogIntroduction?xpage=rename&step=1",

     "doc_hasrights": true,
     "doc_rights_url": "/xwiki/bin/edit/Blog/BlogIntroduction?editor=rights",

     //
     // Cell values
     //
     "doc_title": "First blog post",
     "category": "News",
     "publishDate": 1243846800000
      ...
    },
    ...
  ]
}

See the XWiki.LiveTableResults page which is our current default data source.

It's important that the data source doesn't depend on the JavaScript widget used to implement the live table. If necessary, we can write adapters on the JavaScript side to modify the JSON returned by the data source in order to meet the format expected by the JavaScript widget.

The Live Table Widget

This component is responsible for displaying the live table. It reads the live table configuration from the DOM element being enhanced, fetches the live table data from the server and displays each row by calling the right cell displayer.

Basically this component is responsible for:

  • loading the initial data
  • reloading the data when the user uses the pagination, sorts, filters or deletes rows
  • displaying the rows (delegates to the cell displayers)
  • pagination, sorting, filtering, selection and row actions (e.g. delete)
  • adding new rows
  • editing cells
  • re-ordering columns, removing / hiding columns, adding / showing columns
  • URL hashing (to be able to bookmark the live table state)

This component needs to have at least 2 sub-components:

  • a generic live table / grid widget; for this we should use an existing open-source JavaScript library with WebJar packaging
  • the code that integrates the existing live table widget in XWiki

There are 2 ways in which the live table initialization is triggered:

  • automatic, based on some CSS class, e.g. xwiki-livetable
    • on page load
    • on xwiki:dom:updated event
  • manual, through a jQuery plugin:
    $('selector').livetable({
      // configuration
    });

The live table widget configuration could look like this:

{
 //
 // Modifiable
 //

 // The list of available columns to choose from when adding a column, along with their descriptor.
 // Creating a new column adds a new entry (descriptor) to this list.
 columnDescriptors: [
    {
     // Used when displaying and filtering the live table rows.
     id: 'doc.title',

     // Displayed in the table header.
     name: 'Title',

     // Displayed on hover over the column name, if specified.
     description: '...',

     // Displayed before the column name, if specified.
     icon: '...',

     // The column type, chosen when creating the column. The live table widget uses this only to prefill the column
     // descriptor. When the column is mapped to an xclass property this specifies the property type.
     type: 'String',

     // If not specified then the column is not sortable.
     sortable: true,

     // Used to compute the name of the RequireJS module that will be used to display/edit the values from this column.
     // If not specified then the default 'text' displayer is used.
     displayer: 'link',

     // This is used only by the 'link' displayer (which receives the row data and the column descriptor).
     linkType: '...',

     // Used to compute the name of the RequireJS module that will be used to filter the values from this column.
     // If this is not set or set to false then the column is not filterable.
     filter: 'text',

     // This is used only by the 'text' filter (which receives the column descriptor).
     match: 'prefix',

     // Optional CSS class name to add to the TH element.
     styleName: '...'
    }, {
      id: 'birthdate',
      name: 'Birthdate',
      icon: '...',
      sortable: true,
      displayer: 'date-timeago',
      filter: 'date-range'
    }, {
      ...
    }, {
      id: '_actions',
      name: 'Actions',

      displayer: 'actions',
     // This is used only by the 'actions' displayer (which receives the row data and the column descriptor).
     actions: [
        {id: 'delete', label: 'Delete', icon: '...'},
        ...
      ]
    }
  ],

 // The list of columns to display.
 columns: ['doc.title', 'birthdate', ...],

 // The list of columns to sort on. The sort order can be optionally specified.
 sort: ['birthdate:desc'],

 // The values used to filter the live table rows. The live table filters are prefilled with these values. When a new
 // filter is applied this configuration is updated.
 filterValues: {
    birthdate: ['...', ...],
    ...
  },

 //
 // Read-only
 //

 // The list of known column types. If this is specified then allow the user to choose the column type when creating a
 // column (and prefill the column descriptor with the settings from the column type).
 columnTypes: [
    {id: 'String', name: 'String', icon: '...', sortable: true, displayer: 'text', filter: 'text'},
    {id: 'Date', name: 'Date', icon: '...', sortable: true, displayer: 'date', filter: 'date-range'},
    ...
  ],
 
 // The list of known filters to choose from when creating a new column.
 filters: ['text', 'date-range', 'user', ...],

 // The list of known displayers to choose from when creating a new column
 displayers: ['text', 'html', 'link', 'date', 'date-timeago', 'user', ...],

 // Limit the number of rows displayed.
 limit: 10,

 // The maximum number of page links to display in the pagination.
 maxPages: 10,

 // Used to build the page size drop down.
 pageSizeBounds: {min: 10, max: 100, step: 10},

 // This is used to compute the name of the RequireJS module that will be used to get the live table data. If not
 // specified, a default source is used.
 source: '...',

 // This is used by the live table source module to get the live table data.
 url: '...'
}

If we want to make this generic in order to handle any data representation (table, card, calendar, etc.) then the configuration could looks like this:

{
 //
 // The query
 //

  query: {
   // The list of properties to fetch.
   properties: ['title', 'year', ...],

    source: {
     // This is used to compute the name of the RequireJS module that will be used to get the data. If not specified, a
     // default source is used.
     id: '...',

     // This is a parameter used by the specified source module.
     url: '...'
    },

   // The constraints to apply on the property values. These are hidden constraints that the user cannot change.
   hiddenFilters: {
      year: ['...', ...],
      ...
    },

   // The visible filter values that the user can change. The filters are prefilled with these values. When fetching
   // the data this is merged with the hidden filters (see above).
   filters: {
      year: ['...', ...],
      ...
    },

   // The list of properties to sort on. The sort order can be optionally specified.
   sort: [{
      property: 'birthdate',
      descending: false
    }, ...],

   // Indicates where the current page starts.
   offset: 0,

   // The number of entries to fetch (the page size).
   limit: 10
  },

 //
 // The data
 //

  data: {
   // The total number of entries available (on the server side).
   count: 54,

    entries: [
      {
       // property: value
       title: 'Work from home',
        year: 2020,
        ...
      },
      ...
    ],
  },

 //
 // The meta data (used to control how we interact with the data)
 //

  meta: {
   // Describes the properties that may appear in the data set. This determines the list of known (available)
   // properties. Creating new properties, removing existing properties as well as editing the property descriptor
   // should be done through this array.
   propertyDescriptors: [
      {
       // Identifies the property that this descriptor corresponds to.
       id: 'title',

       // The property name. Could be displayed before the property value.
       name: 'Title',

       // Could be displayed when hovering the property name.
       description: '...',

       // Could be displayed before the property name, if specified.
       icon: '...',

       // The property type, selected when creating the property. It is used to prefill the property descriptor.
       // Could be mapped to an xclass property type.
       type: 'string',

       // Whether the user can sort on this property or not. If not specified then the property is not sortable.
       sortable: true,

       // Displayer configuration.
       displayer: {
         // Used to compute the name of the RequireJS module that will be used to display/edit the values from this
         // property. If not specified then the default 'text' displayer is used.
         id: 'link',

         // This is used only by the 'link' displayer (which receives the data entry and the property descriptor).
         linkType: '...'
        },

       // Filter configuration.
       filter: {
         // Used to compute the name of the RequireJS module that will be used to filter the values from this property.
         // If this is not set or set to false then the property is not filterable.
         id: 'text',

         // This is used only by the 'text' filter (which receives the property descriptor).
         match: 'prefix'
        },

       // Optional CSS class name to add to the HTML element used to display this property.
       styleName: '...'
      }
    ],

   // The list of known property types. When creating a new property the user can select from this list and the
   // property descriptor will be prefilled based on the selected property type.
   propertyTypes: [
      {id: 'string', name: 'String', icon: '...', sortable: true, displayer: {...}, filter: {...}},
      ...
    ],

   // The list of known filters to choose from when editing the property descriptor.
   filters: [
      {id: 'text', ...},
      {id: 'date-range', ...},
      {id: 'suggest', ...},
      ...
    ],

   // The list of known property displayers to choose from when editing the property descriptor.
   displayers: [
      {id: 'text', ...},
      {id: 'html', ...},
      {id: 'link', ...},
      {id: 'actions', ...},
      ...
    ],

   // Configure the pagination display.
   pagination: {
     // The maximum number of page links to display in the pagination.
     maxShownPages: 10,

     // Used to build the page size drop down that allows the user to change the number of entries displayed per page.
     pageSizeBounds: {min: 10, max: 100, step: 10}
    }
  }
}

The live table widget needs to trigger some events. See the current list of live table events for details.

  • when the live table is ready (initialized)
  • whenever the live table rows are updated (due to pagination, sorting, filtering, etc.)
  • after a row is displayed
  • when a column is created, added, removed, edited or deleted
  • when a row is created

The live table widget is currently implemented by livetable.js.

Filters

The filter JavaScript modules help the user pick the live table filter values. There are 2 types of (visual) live table filters:

  • column filters: displayed below the column header, they filter the values from that column
  • external filters: displayed outside the live table (e.g. before), they are not bound to a column and they filter the live table rows based on values that are not necessarily displayed

We need at least the following pickers for column filters:

  • date range: allow the user to select a date range using for instance 2 calendars
  • list with multiple selection: allow the user to filter the column by selecting one or multiple values from a limited list of known values
  • suggest picker: help the user select a value from an unlimited list of values by suggesting the existing values

We need to support at least the following external filters:

  • tag cloud: filters the live table rows based on the tags associated to wiki pages (when the live table rows represent wiki pages)

The live table widget shouldn't be aware of the filter pickers and external filters. The filter picker modules are included by the Velocity macro based on the live table configuration (e.g. the date range picker is included only if there is a date column) and they enhance the live table filters using the events triggered by the live table widget.

Displayers

The displayer modules are responsible for displaying each cell, both when viewing and editing the live table data. They could be implemented as RequireJS modules and the live table configuration would specify which displayer module to use for each column.

define('xwiki-livetable-displayer-date', [...], function(...) {
 return {
    view: function(column, rowData, tableConfig) {
      ...
     return domElement;
    },
    edit: function(column, rowData, tableConfig) {
      ...
     return domElement;
    }
  }
});

The following displayers are needed:

  • default: simply display the value for view and a plain text input for edit
  • date: support time-ago format and a date picker for edit
  • page: link to page on view and page picker when editing
  • user: user avatar and link to user profile on view and user picker on edit

Synchronization

This module is responsible for updating the live table when the data is modified on the server. There are two types of synchronization that we can achieve:

  • Closed synchronization (less complex)
    • Doesn't depend on the data source
    • Only the changes done within the real-time session are propagated (e.g. only the changes done through the live table)
  • Open synchronization (more complex)
    • Depends on the data source (specific change listeners have to be written for each data source)
    • Any change done to the data source is propagated (e.g. the live table is updated when one of the pages that is listed is modified outside of the live table)

For the scope of this proposal we will focus only on closed synchronization.

User Cases

The brand new feature of the Livetable 2.0 is real-time editing.

The entries can be directly edited from the Livetable fields. The modified objects are updated on the server, and in all the other instances of the Livetables displaying this data, without needed to reload the page.

If two user are editing an entry at the same time, existing conflicts should be resolved automatically.

User Case 1:

user1 and user2 are viewing the results of the same Livetable

  • user1 modify a property of an object directly from the Livetable
  • the object is modified on the server
  • user2 gets its own Livetable instantly updated

Similarly, if the entry is modified from its edit page, it has to update the opened Livetables too.

User Case 2:

user1 is browsing a Livetable, while user2 is currently editing an object in its edit page

  • user1 modify a property in the same object directly from the Livetable
  • the object is modified on the server
  • user2 do not get the update about the modification, as the page is static
  • user2 modify a property in the object and save the changes
  • the object is modified on the server
  • user1 gets its Livetable instantly updated, and his modification gets overridden

Moreover, an object can be displayed in Livetables other than the one of its application. Any Livetable using an object should watch for any updates of this object.

User Case 3:

user1, user2 and user3 are viewing the results of different Livetables

  • user1 modify a property of an object directly from the Livetable
  • the object is modified on the server
  • user2 have this object on its Livetable, so it gets instantly updated
  • user3 does not have this object on its Livetable, so nothing changed for him

The User Case 3 is more complicated to set up, because we have to watch for the update of any entries of the Livetable, and no longer of the Livetable itself.

Beyond the real-time Livetable

This project is the first step of creating a real-time editable wiki.

The end-goal would be to update in real-time not only the Livetables, but also any occurrence of an object in the wiki.

User Case 4:

user1, is viewing the results a Livetable, and user2 is browsing some wiki pages

  • user1 modify a property of an object directly from the Livetable
  • the object is modified on the server
  • user2 have that property of the object displayed on its page, and it gets updated

This is not part of the project, but this could be taken in account, to built code that could extend to this purpose in the future.

The Livetable Implementation

Logic vs Layout script

A good feature of the new Livetable would be to support different layouts for displaying the data to the user. The table layout is obviously the first one we need to implement, but we could also add other ones like cards layout or calendar layout when the first one is done.

If we want to implement such a feature, we need to think our code in two separated layers:

  • the layout script
    • only display what it receives from the logic script
    • listen to user interaction, and notify the logic script appropriately (e.g. the user click on the first column header, it tells the logic part to sort the column 1 in reverse direction)
    • there can be different layout scripts: table, cards, calendar, timeline, ...
  • the logic script
    • handle all the logical operations: filtering, sorting, pagination, URL hash, ...
    • intermediate between the macro and the layout script
    • when it received a change notification from the layout script, it apply the changes and fetch the new data from the server accordingly, then pass the new data to the display layout
    • works like an API, can communicate with other XWiki widgets, is easily extendable

As the user may want to switch between the Livetable available layouts, the logic script could dynamically import the layout scripts it needs. The available layouts should be specified in the macros.

Editing mode vs Designing mode

The Livetable should be able to handle two modes:

Edit mode (local)

The user can modify the data in existing rows and add new rows.

The configuration modifications are local, and can be saved with the URL hash. This includes:

  • switching layout
  • filtering, sorting, columns visibility
  • other layout specific configuration

The user cannot perform actions locally on the data structure, like adding properties locally (this would make no sense).

Design mode (global)

The user can still modify the data.

The configuration modifications are shared between users in design mode, but are not pushed to the server directly. It allows users in edit mode to not receive the modifications, and be able to locally view / edit the data without being bothered by the structured being changed at the same time.

The layout configuration are global, and will be saved as default configuration. This includes:

  • choosing default layout
  • choosing default filters, sort, columns visibility (and )other layout specific configuration
  • adding / deleted properties
  • changing type of existing properties

Users with the appropriate rights can push the new Livetable config to the server. This will update the Livetable struture in edit mode too (or show a notification asking to refresh the Livetable in oreder to see the new structure, so that users do not lose their current local config right away).

The table layout technologies

For each layout, we might need to use a library to help us displaying the data.

In this proposal, we will focus only on the table layout, as it was only the first one required.

Solution 1: Using a table library

We could use a JavaScript library to create HTML tables from existing data. This would simplify and speed up the development of the new Livetable.

However, the library will probably not handle the user interactions around the intended features (sorting, filtering, modifying...) the way we want. We will have to rewrite those functionalities in the way we want them to behave. This might become a problem if there are too many functionalities we have to rewrite. We will end up with a whole library where we only use the displaying part of it, and a lot of functions overriding the rest of the existing functionalities.

What we could do to solve these problems is to create a facade / adapter object that uses the library functionalities, and adapt them to our needs. Also, in this way the implementation will stay rather independent from the chosen library, and we will be able to upgrade or change the library in the future by just updating the adapter code. This would be great for maintainability in the future years

Several libraries could serve this purpose:

Tabulator.js

pros:

  • development very active (new release every 2~3 months)
  • modular implementation: we can easily overwrite and adapt the existing functionalities
  • already comes with the intended features: display / editing cells by column type. We can add custom formatters / editores for the types we want.

cons:

  • does not natively support IE11, polyfills needed
  • not very responsive on small layout (can only hide or collapse columns, but collapsed columns cannot be editable through Tabulator native edit system)
  • didn't find how to edit cells with dblclick instead of single click
  • native column reorder is a bit buggy

Datatables

pros:

  • development still active
  • lightweight and configurable
  • very easy to add custom formatters for columns

cons:

  • does not support editable cells by default
  • not really responsive (has an extension to collapse columns but it looks awful, however we could create our own responsive extension)
  • column formatters work with outerHTML and not HTML elements, so they would be more limited
  • if a property does not exists in an entry (i.e the property is not empty but undefined), it window.alert an error by default (we can override that but... why?)

Others solutions are available, but they are less configurable, no longer in development, or need a commercial license.

Solution 2: Using a framework

We could also re-write entirely the Livetable component by ourselves. As opposed to the solution 1, this would imply a little more development time, but greater flexibility and maintainability. Instead of using existing code that we have to adapt, we would already build code that meet our needs in the first place. Furthermore, if the layout script only need to display data and handle user interaction, it might not be too complex to create a table from scratch.

In this case, we should use a framework to fasten the development, by automatically binding the HTML table to the JSON object, and helping with event handling.

Vue.js would be a great solution, it is versatile and easy to set up. With this framework we would be able separate our Livetable in smaller components that could communicate with each other. We could think of components like table rows, table headers, or column filters. With this technique, the code would become more understandable and maintainable in the future.

We will have to add an event adapter for Vuejs so that it can communicate with the rest of the page.

We can also use some utility libraries to help us during the development, like libraries for drag-and-drop, custom selects, or menus.

Implementing existing features

The features of the first version should still be present in the new one.

These features include:

  • Filtering entries
  • Sorting entries
  • Pagination
  • Actions to view, edit or remove the entries
  • Responsiveness

These features has to be implemented differently from the first version, to allow more flexibility.

For example, the Livetable of Notion.so allows users to perform complex filtering and multi-columns sorting. However, theses features are not always displayed to the user, and are hidden under a sub-menu of each feature respectively.

It would be nice to find a compromise between the complexity of the feature and its easiness to use for the user, and to have the options visible right away.

Notion also allows the user to modify not only the data but also the structure of the table (adding / removing columns, changing type of column, ...). When displaying a Livetable on a page this is not desired, the user should only be able to modify permanently the data. What we could do is let the user perform several operations on the columns only on its local instance of the Livetable, like re-ordering, showing or hiding them.

As several display layout would be available (table, cards, ...), we also need to consider if we want these features to be implemented in the same way for each layout. If it's the case, that means less development time, and more coherence between the layouts. However, certain layout could have a better suited implementation that would be more intuitive for the user.

Sorting

For the moment, we can only sort the table according to one column.

Implementing a multi-sort intuitive for the user is quite complicated. Most of existing sortable tables does not implement multi-sorting.

For those implemented such a system, they came with different implementations:

  • keeping the previous sorts order for sub-level sorting (Google Sheets)
    • pros: really simple, no wizard or sub-menu needed
    • cons: not practical at all for the user experience
  • using a wizard (Libre Office): we can choose which column correspond to which sort level
    • pros: easy to understand and perform
    • cons: not easily accessible, not easily readable (the user cannot see with one glance how to column are sorting the table)
  • using a sub-menu (Notion.so): we access in a sub-menu a sortable list of the columns used for the multi-sort
    • pros: easy to perform, a bit more accessible than using a wizard
    • cons: still not accessible / readable right away

Each of the described solutions makes compromises, and come with pros and cons.

Moreover, in most of the cases, we don't have any indication about which column correspond to which level of sorting.

In our final solution, we will need to get the sort-system as accessible and readable as possible.

Solution 1

This is our custom solution for multi-sorting column.

We can add a number next to the triangle indicating the level of sorting:

image-20200511112549-12.png

(Or with color to be more distinguishable from each other)

image-20200511160945-2.png

Not all the columns have to sort the data, here only Title and Date are used (unused columns shows a ghost triangle on hover):

When we left-click on the icon (or the whole title):

  • if the column was already sorting (at any level), it changes the direction of the sort (asc / desc)
  • else, it became the new level-1 sort

When we right-click on the icon, a sub-menu appears to let us choose the level of sort we want:

image-20200511161346-4.png

  • if we click on the current sort level, it changes its direction
  • else, it overrides the existing sort of the chosen level

Only the useful levels are displayed in the sub-menu: an already sorting column will not show lower level sorts, and a non-sorting column will show levels until the lowest level possible in the current configuration.

Pros:

  • easy to visualize how the columns are sorted
  • quick to modify the different sort levels

Cons:

  • right-clicking might not be intuitive for the user, and he might never know it's possible
  • the colors of the sort levels, even though they are common, might not looks good in some themes, or might not be wanted by the user (solution: every level stays black)

Solution 2

This second solution is a variation of the first one. Instead of displaying the direction and the level together, we can display both information separately:

image-20200515112513-1.png

When we hover a title with no sort on it:

image-20200515112904-2.png

When we click on the title or the triangle icon: same behavior than the left-click of solution 1

When we click on the "+" button: same behavior than the right-click of solution 1

Pros:

  • same pros than solution 1
  • no more unintuitive right click
  • as the level is displayed on its own, we no longer need to impose the colors to ease the distinction

Cons:

  • it might not be clear that the plus sign refers to creating a new level of sort

(Note: with this solution we could still keep the behavior of the right-click as an alternative way to open the sort level sub-menu.)

Solution 3

We create a sub-menu above the table where we can specify the order of the columns in the multi-sort (like in Notion). This solution would work on any layout.

Pros:

  • keeps the UI light
  • works on any layout

Cons:

  • functionality hidden behind a menu: we are adding extra steps to sort the table
  • we can't know with a glance how the columns are sorted

We could still display the triangle icon in the table header to indicate that the column is currently sorting the table. Clicking on it would open the sort menu.

Filtering

For the moment, the filter only try to see if the rows match the specified text. It is not possible to check for the inequality of a number, nor for the range of a date for example.

It would be nice to have more complex filtering options, like: "=", "≠", "≤", "≥", "between", ... Specific operators should be added according to the property type (date, list, ...).

Also, for now it's not possible to combine filters in another way than a "AND" logical operator between the columns. A solution could be implemented to be able to combine different columns with a "OR" operator (e.g. name="jean" OR age≥25), or to be able to combine different filters in the same column (e.g. age≤20 OR age≥50) The combination of "AND" and "OR" operators should be kept intuitive though.

Solution 1

For now, we can only have 1 filter for each column, and we can only match equality.

What we can do first is to append the operator before the input, with the ability to change it by clicking on it:

image-20200511114754-15.png

We can also allow multiple filters by adding an action to create more :

image-20200513100959-1.png

Clicking "add filter" will automatically focus the created input.

When there is no filter, there is only the "add filter" action, so that the UI stays light.

From the second filter, a new button is added allowing to choose between the "AND" and the "OR" operators. The logic operator has to stay the same for all filters in the same column. For instance if "AND" is chosen, all the filters of the column will be combined using the "AND" operator.

With this current design, there is no other way to change the way columns are combined together. For now, all the columns are combined with the "AND" operator. In any way, the operator should also stay the same between each columns to keep the behavior understandable.

Here is a test of the design on large layout with a lot of filters:

image-20200513104407-1.png

and on narrow layout:

image-20200513104531-2.png

On the narrow layout, the fields are becoming too small, and the result becomes quite heavy (we can work on the design of the inputs to clean the design a bit more).

In addition to that, the filters take much more width than the column content, this is really showing up in number columns like the "Age" one for instance. This might become a problem for large Livetable with a lot of columns.

However, on average the user is not going to create 3 filters for each columns, and not filters every columns. As the UI complexity is proportional to the filtering complexity, the UI should stay light most of the time.

Pros:

  • unlimited number of filters for each column
  • filters directly accessible: the user can quickly filter, and knows instantly what's being displayed
  • straightforward

Cons:

  • we can only use the "AND" operator between columns with this current design.
  • the filters can take much more width than the column content
  • not lightweight at all

Solution 2

We create a sub-menu above the table where we can specify the filters we want to apply to each columns (as Notion does). This solution would work on any layout.

Pros:

  • unlimited number of filters for each column
  • as opposed to the solution 1, the width of the filters is not a problem anymore
  • keeps the UI light
  • works on any layout

Cons:

  • filters not accessible directly: we are adding an extra step to access the filters
  • filters not viewable by default: we can't know with a glance how the columns are filtered

Filtering by Tags

There is currently a way to filter by tags, from the outside of the Livetable. As the script layer should behave like an API, it should be easy to communicate with it.

Global search

We can also add a global search input above the Livetable, that is going to search if the input text can be found anywhere in the entries of the data source.

Any entries containing the query text in one of their properties will be returned. This filtering system should combine with the normal one, with an "AND" operator.

Filtering operators

Filtering a string is not the same that filtering a number, a date, or a list of values. Thus, we should not provide the same actions for all type of filters, but adapt according to the filter type.

When adding a filter on a column, the program has to do 2 things:

  • create the operator select, corresponding to the column type
  • create the input along with its picker, corresponding to the column type, and the selected operator if needed

There can be different pickers for a same property type. For instance, a "Number" type can be associated to the default number input, or to a custom rating select showing stars. Even if the rating select displays stars to the user, it will return a integer (between 1 and 5) at the end.

Ideally, the operators select should not be aware of anything but the property type. However, as the data source may not come with code written for certain operators for the query creation, and as there is no fallback to a default query creation based on the type of the property, we cannot assume that.

Instead, what we have to do is to disable the operators that are not supported by the data source, indicated in the Livetable JSON configuration object.

For each property type, the default operators should be:

  • Text
    • "Equals"
    • "Dos not Equals"
    • "Starts with"
    • "Ends with"
    • "Contains" (default)
    • "Does not Contains"
    • "Before"
    • "After"
    • "Like": allows to perform regex-like filtering for advanced users
    • "Is empty"
    • "Is not empty"
  • Number
    • "=" (default), "≠", "<", "≤", ">", "≥"
    • "between": shorthand for "≥ filter1 AND ≤ filter2", displays two inputs instead of one
  • Date
    • "Is" / "=" (default)
    • "Is not" / "≠"
    • "Before" / "≤"
    • "After" / "≥"
    • "between": shorthand for "After filter1 AND Before filter2", displays two inputs instead of one
    • "Is empty"
    • "Is not empty"
  • List (unique values)
    • "Is" (default)
    • "Is not"
    • "Value In": shorthand for "Is filter OR Is filter OR ...", displays a select
    • "Value not in": shorthand for "Is not filter OR Is not filter OR ...", displays a select
    • "Is empty"
    • "Is not empty"
  • List (multiple values)
    • "Contains" (default)
    • "Does not contains"
    • "Values In": shorthand for "Contains filter OR Contains filter OR ...", displays a select
    • "Values Not in": shorthand for "Does not contains filter OR Does not contains filter OR ...", displays a select
    • "Is empty"
    • "Is not empty"

Some notes:

  • the filters are case aware: lowercase matches both lowercase and uppercase, while uppercase only matches uppercase
  • the operators written in words can take up too much space depending the chosen implementation. This would be the case in the Solution 1.

Pagination

The pagination is already functional and nothing but its design might be changed.

We could also implement an infinite scroll feature that the user could choose instead of the pagination system in the table options. However, this would not be great for every cases, as there might be some content under the Livetable that could not be accessed anymore as the table is continuously expanding. The scroll feature could be done inside the Livetable to fix this issue.

Similarly, we could display a "load more" button at the bottom of the Livetable so that the user manually fetches new entries.

Data export

In the new design, we need to keep the feature allowing the user to export the content of the Livetable to csv format.

This is not the priority, be we could add other formats, like excel or json.

Implementing new features

Editing cells

If the user single-click on a cell, the outline of the cell changes color, showing that the cell is currently focused. A tooltip could be displayed on hover telling to double-click in order to edit. The current focused cell is only visible by the user and is not shared with other viewers of the Livetable.

If the user double-click on a cell, the cell will go into edit mode. The displayer module will be replaced by its corresponding editor module to allow the user updating the value. When the user goes out of focus or the user press "Enter", the changes are kept, and the displayer is used again to display the new value. If the user press "Escape" instead, the cell goes back to it initial value. If the user presses "Tab" while editing a cell, the changes are kept and the next cell is focused in edit mode.

Adding a new row

The user should be able to easily insert a new row in the table.

However, as the Livetable does not display all the properties of the pages, we cannot just insert a new row with all its cells editable, as the created page would be incomplete.

We can create a button above the Livetable to insert a new row in the Livetable.

When the user click on it, the action that follows could be either of:

Solution 1

We simply redirect the user toward the add-new-entry page.

Pros:

  • cannot be simpler

Cons:

  • we leave the Livetable page to create a new entry
  • there are now two buttons with the same effect on the page

Solution 2

We display a wizard in a modal allowing the user to fill all the properties of the new entry. This solution is implemented in Notion.

Pros:

  • we do not leave the page

Cons:

  • way more complicated than solution 1
  • we have to create another wizard doing the same thing than the one in the add-new-entry page

Solution 3

We add a new empty row at the bottom of the Livetable, only if all the properties are displayed to the user.

Pros:

  • really intuitive to add new data for the user (coherent with the editing system)

Cons:

  • if all properties are not displayed, we cannot use this system
  • and using two different systems to add new rows could be confusing for the user

Handling rows in the wrong page

When inserting a new row or modifying an existing one, there is a possibility that the row has to be displayed in a different page that the current one, because of the Livetable configuration (sorting, filtering, ...). When this happens, we cannot just move the row to the new page it belongs, as it would be confusing for the user (the row would just disappear with no indication).

To avoid that, we can hold the modified row in the current page with a different state, shown by a different display style. In front of the row, we can display an icon representing an "i" in a circle (for "information"), that explains on hover the current state of the row, and that it will be back to normal on the next user action (change sort, filters, modify another row...).

Selecting rows and Batch operations

This feature has been asked by many users: it would be nice to be able to select rows in the table, and to execute an action for all of them.

The action could be: deleting the entry, changing the rights, modifying value...

This would not be complicated to set up: we only need to add column on the left of the table where we display a checkbox for each row. In the column header, there is a checkbox that allow to quickly check / uncheck all the present rows in the table.

Once we select at least one row, a menu appears (or become enabled) above the table, and let the user perform the desired action.

Columns operations

There are several operations on the columns the user might consider, like reordering or hiding some of them. These action would only affect the user display and would have no effect on the server or the other Livetables.

We can include these changes in the URL hash (automatically or by user action) so that the user can keep the current state of his table, or share it with other people.

Moreover, we can think of a system that allow the user to create a new page based on the current Livetable configuration.

Showing / Hiding columns

We can create a sub-menu above the table, showing all the columns the Livetable can display, each column associated with a checkbox allowing the user to toggle its visibility.

If some columns are hidden in the Livetable, we should display an icon in the sub-menu title to indicate that not all the columns are shown. The icon could be an exclamation mark or an eye (or both).

Reordering

The easiest way for the user to reorder columns is by drag and dropping them at the place they want.

We could also allow the columns listed in the sub-menu above the table to be order-able.

Resizing the columns

This is not the priority but if anyone wants this feature, it could be implemented to the Livetable.

Other concerns

  • The new Livetable should be compatible and adaptable with the current styling configuration of the wiki.
  • It should also respect the accessibility specifications:
    • be a valid ARIA table
    • be usable with the keyboard only (through tabindex)
  • The data received from the server should be cached to alleviate the workload of the server when going back to a previously browsed Livetable page.

 

Tags:
    

Get Connected