Wiki source code of RealtimeLivetable

Last modified by Vincent Massol on 2024/02/26 17:54

Show last authors
1 This page lists the specifications of the new live table.
2
3 {{toc depth="3"/}}
4
5 = Functional Requirements =
6
7 R1: **Load**
8
9 * Load the data asynchronously (don't block the page load)
10 * Show a loading animation
11
12 R2: **Display**
13
14 * Display the data as a table (with visible table header, rows and columns)
15 * Responsive display on small screens
16 * Be able to mark columns that can be hidden when there's not enough horizontal space
17 * Display the data as a grid (card layout)
18
19 R3: **Pagination**
20
21 * Load only a limited set of rows (one page)
22 * Show the total number rows
23 * Be able to change the number of rows displayed per page (10, 15, etc.)
24 * Show page numbers vs. show only First / Next / Previous / Last links
25
26 R4: **Sort**
27
28 * Be able to sort the live table rows by a single column, both ascending and descending
29 * Be able to sort on multiple columns
30 * Indicate the sort column and the sort order (ascending / descending)
31
32 R5: **Filter**
33
34 * Be able to filter the live table rows by one or multiple columns
35 * Indicate the columns with active filters
36 * Have a single filter/search input vs. have a filter per column
37 * When filtering by multiple columns, use the **AND** operator (match all)
38 * Be able to specify multiple constraints for a single column, and allow to choose whether to apply all (AND) or any (OR)
39 * Be able to filter by exact value, partial value, prefix, suffix, less than, greater than
40
41 R6: **Selection**
42
43 * Be able to select live table entries one by one, even from different pages
44 * Be able to select all / none of the entries from the current page (loaded entries)
45 * Be able to select all / none of the entire entries (including entries that haven't been loaded yet)
46
47 R7: **Actions**
48
49 * Be able to perform actions that target a single row (e.g. view details, delete a row)
50 * Be able to perform **batch** actions that affect the current selection of rows
51 * Standard actions: view, edit, delete, copy, export
52
53 R8: **Bookmark**
54
55 * Be able to bookmark or share the current state of the live table (pagination, sort, filters, selection) with other users (through the URL)
56 * Be able to obtain the wiki syntax needed to recreate the current state of the live table (to copy & paste in another wiki page)
57
58 R9: **Synchronization**
59
60 * Be able to see in real-time the changes done by other users through the **same** live table (e.g. if another user deletes a row then update my live table to remove that row)
61
62 R10: **In-place edit**
63
64 * Be able to **edit** the live table data in-place
65 * Be able to **add** a new live table entry from the live table
66
67 = Architecture =
68
69 The following components are involved:
70
71 * A **Velocity macro** that loads the required JavaScript modules and CSS stylesheets and outputs the HTML placeholder
72 * A **wiki syntax macro** that calls the Velocity macro
73 * A generic/reusable **data source** that retrieves the data from wiki pages and their objects, and that can be extended or customized
74 * A (responsive) **stylesheet**
75 * Multiple JavaScript modules:
76 ** **Core**: looks for the HTML placeholder in the DOM and enhances it, then fetches the data and fills the table
77 ** **Filters**: separate modules that enhance the live table filters (e.g. help the user pick a date range to filter a date column)
78 ** **Displayers**: separate modules that handle the display of the data inside the live table (e.g. user displayer, date displayer, etc.)
79 ** **Synchronization**: the module that handles the live table synchronization (real-time updates)
80
81 Would be nice to also have a wizard to help the users generate the live table configuration.
82
83 Let's see how the listed components interact with each other:
84
85 * The user inserts the live table **wiki syntax macro** in a page.
86 * When the page is rendered the wiki syntax macro is executed (on the server).
87 * The wiki syntax macro then
88 ** reads the live table **configuration** (from its macro parameters, or from its macro content or from another source)
89 ** and calls the live table Velocity macro, passing the configuration.
90 * The **Velocity macro** (on the server)
91 ** includes the required JavaScript modules (core, filters, displayers, synchronization), depending on the live table configuration,
92 ** and the stylesheets,
93 ** then outputs the HTML placeholder, also passing the live table configuration for the client side using HTML data attributes
94 * The browser parses the page HTML and loads the JavaScript and CSS resources
95 * When the live table **core JavaScript module** is loaded
96 ** it looks for the HTML placeholder in the DOM and reads the live table configuration
97 ** enhances the HTML structure
98 ** makes and HTTP request to fetch the live table data (using the **data source** specified in the live table configuration)
99 *** (on the server, the data source receives the HTTP request and executes the necessary database queries to gather the data)
100 ** fires the events needed to trigger the initialization of the live table **filters** (which are handled by separate JavaScript modules)
101 ** fires the events needed to trigger the display of the live table data (based on the configuration of each column)
102 ** initializes the **real-time** session (synchronization)
103
104 {{info}}
105 **TODO**: Would be nice to have a diagram to view the interaction.
106 {{/info}}
107
108 Let's look at each component in detail.
109
110 == Wiki Syntax Macro ==
111
112 {{code language="none"}}
113 {{livetable ... /}}
114 {{/code}}
115
116 The wiki syntax macro is meant to be used by both **simple users** and advanced users so it needs to have a good balance between easy to understand parameters and more advanced parameters that have good default values. The live table settings could be read from:
117
118 * macro parameters and/or the macro content (= in-line configuration source)
119 * an external configuration source (e.g. from some dedicated objects attached to a wiki page)
120 * both, using the external configuration source as a fallback
121
122 Here's a list of common live table settings that the user should be able to specify when calling the wiki syntax macro:
123
124 * the **data source**, can be specified as:
125 ** the reference of a wiki page that holds a class definition (the default / generic data source is used)
126 ** the reference of a wiki page that implements a custom data source (could extend the default data source)
127 ** a URL for a custom data source
128 * the **list of columns**
129 * the columns to **sort** the live table on initially, including the sort **order** (ascending / descending)
130 * the maximum **number of rows** to display per page
131 * the **translation** key prefix
132 * **filter values**
133 ** the values used to pre-fill the live table (visible) filters
134 ** and the values passed to the data source in order to restrict the data set (hidden filters)
135
136 Based on this the wiki syntax macro could have the following parameters:
137
138 {{code language="none"}}
139 {{livetable
140
141 ## Optional parameters that map to the corresponding HTML attributes, helping developers customize the live table
142 ## styles and behavior.
143 id="foo"
144 class="bar"
145
146 ## Optional parameter that specifies the source of the live table data. The parameter value is a component hint. A
147 ## default implementation is provided (mapped to the current XWiki.LiveTableResults page).
148 source="default"
149
150 ## Optional source parameters, used mainly to apply hidden filters (i.e. filters that cannot be removed from the UI).
151 sourceParameters="className=XWiki.XWikiUsers&location=Users"
152
153 ## Optional comma-separated list of columns. The default list of columns is specified by the data source.
154 columns="one,two,three"
155
156 ## Optional comma-separated list of columns to sort on. Sort order can be specified.
157 sort="one,two:desc"
158
159 ## Optionally initialize the visible live table filters (the users can change these values from the UI).
160 filter="one=blue&three=pending"
161
162 ## Optionally limit the number of live table rows displayed.
163 limit="10"
164
165 ## Optional translation key prefix. Should be used only to overwrite the default labels.
166 translationPrefix="foo.livetable."
167
168 /}}
169 {{/code}}
170
171 Check [[this page>>Design.LiveTable20Macro]] for a related proposal. Note that column properties (e.g. whether the column is filterable or sortable) are not included in the list of macro parameters because they should be specified by the data source.
172
173 The wiki syntax macro can be used directly in wiki syntax (e.g. from the Wiki editor) but it can also be inserted from the WYSIWYG editor, using the Insert Macro wizard. We need to make sure the macro parameters are easy to set by simple users and for this we need to provide pickers (e.g. a page / xclass picker for the data source).
174
175 The wiki syntax macro serves as a wrapper (facade) for the Velocity macro, so it doesn't know and it doesn't care how the live table is implemented. See the [[Documents Macro>>extensions:Extension.Documents Macro]] for an example of such a wiki syntax macro (the code can be found in the ##XWiki.DocumentsMacro## wiki page).
176
177 == Velocity Macro ==
178
179 {{code language="none"}}
180 #livetable('foo' $columns $columnsProperties $options)
181 {{/code}}
182
183 This is meant to be used only by developers. All the needed information should be passed through macro parameters. The goal of this macro is to include the required JavaScript and CSS resources and to output the live table HTML placeholder, based on the provided live table configuration. It must not depend on wiki pages, because we must be able to use this macro on an empty wiki, from a Velocity template.
184
185 The Velocity macro is aware of the JavaScript widget used to implement the live table and the HTML output it generates might have to be specific to this widget (to meet its expectations). So changing the widget used to implement the live table will require updating the Velocity macro (but should not require an update of the wiki syntax macro or the data source). The Velocity macro is independent of the data source specified in the live table configuration.
186
187 See the existing [[live table Velocity macro>>extensions:Extension.Livetable Macro]] for more details.
188
189 == Data Source ==
190
191 We have to provide a default data source that retrieves data from wiki pages and the objects attached to them. This default data source is generic and can be parameterized (the users can pass configuration parameters through the live table configuration). Developers can extend the default data source or write their custom data sources. The data returned by the data source should include:
192
193 * the number of rows available on the server (needed for pagination)
194 * the offset and limit used (for pagination)
195 * a list of rows (the data to display on each table row)
196 ** something to identify the row (we need this for synchronization most probably)
197 ** access rights (whether the current user can edit, delete, etc. this row)
198 ** URLs of supported actions (edit, delete, copy, etc.)
199 ** the list of cells (the data to display on each cell)
200 *** additional information that may be needed to display the table cell (e.g. data type, pretty name, etc.)
201
202 The JSON currently looks like this:
203
204 {{code language="js"}}
205 {
206 "totalrows": 35,
207 "returnedrows": 15,
208 "offset": 1,
209 "rows": [
210 {
211 //
212 // Identify the row
213 //
214 "doc_wiki": "xwiki",
215 "doc_fullName": "Blog.BlogIntroduction",
216
217 //
218 // Actions & access rights
219 //
220 "doc_viewable": true,
221 "doc_url": "/xwiki/bin/view/Blog/BlogIntroduction",
222
223 "doc_hasadmin": true,
224
225 "doc_hasedit": true,
226 "doc_edit_url": "/xwiki/bin/edit/Blog/BlogIntroduction",
227
228 "doc_hasdelete": true,
229 "doc_delete_url": "/xwiki/bin/delete/Blog/BlogIntroduction",
230
231 "doc_hascopy": true,
232 "doc_copy_url": "/xwiki/bin/view/Blog/BlogIntroduction?xpage=copy",
233
234 "doc_hasrename": true,
235 "doc_rename_url": "/xwiki/bin/view/Blog/BlogIntroduction?xpage=rename&step=1",
236
237 "doc_hasrights": true,
238 "doc_rights_url": "/xwiki/bin/edit/Blog/BlogIntroduction?editor=rights",
239
240 //
241 // Cell values
242 //
243 "doc_title": "First blog post",
244 "category": "News",
245 "publishDate": 1243846800000
246 ...
247 },
248 ...
249 ]
250 }
251 {{/code}}
252
253 See the [[XWiki.LiveTableResults>>https://github.com/xwiki/xwiki-platform/blob/xwiki-platform-12.3/xwiki-platform-core/xwiki-platform-livetable/xwiki-platform-livetable-ui/src/main/resources/XWiki/LiveTableResults.xml]] page which is our current default data source.
254
255 It's important that the data source doesn't depend on the JavaScript widget used to implement the live table. If necessary, we can write adapters on the JavaScript side to modify the JSON returned by the data source in order to meet the format expected by the JavaScript widget.
256
257 == The Live Table Widget ==
258
259 This component is responsible for displaying the live table. It reads the live table configuration from the DOM element being enhanced, fetches the live table data from the server and displays each row by calling the right cell displayer.
260
261 Basically this component is responsible for:
262
263 * loading the initial data
264 * reloading the data when the user uses the pagination, sorts, filters or deletes rows
265 * displaying the rows (delegates to the cell displayers)
266 * pagination, sorting, filtering, selection and row actions (e.g. delete)
267 * adding new rows
268 * editing cells
269 * re-ordering columns, removing / hiding columns, adding / showing columns
270 * URL hashing (to be able to bookmark the live table state)
271
272 This component needs to have at least 2 sub-components:
273
274 * a generic live table / grid widget; for this we should use an **existing** open-source JavaScript library with **WebJar** packaging
275 * the code that integrates the existing live table widget in XWiki
276
277 There are 2 ways in which the live table initialization is triggered:
278
279 * automatic, based on some CSS class, e.g. ##xwiki-livetable##
280 ** on page load
281 ** on ##xwiki:dom:updated## event
282 * manual, through a jQuery plugin:(((
283 {{code language="none"}}
284 $('selector').livetable({
285 // configuration
286 });
287 {{/code}}
288 )))
289
290 The live table widget configuration could look like this:
291
292 {{code language="js"}}
293 {
294 //
295 // Modifiable
296 //
297
298 // The list of available columns to choose from when adding a column, along with their descriptor.
299 // Creating a new column adds a new entry (descriptor) to this list.
300 columnDescriptors: [
301 {
302 // Used when displaying and filtering the live table rows.
303 id: 'doc.title',
304
305 // Displayed in the table header.
306 name: 'Title',
307
308 // Displayed on hover over the column name, if specified.
309 description: '...',
310
311 // Displayed before the column name, if specified.
312 icon: '...',
313
314 // The column type, chosen when creating the column. The live table widget uses this only to prefill the column
315 // descriptor. When the column is mapped to an xclass property this specifies the property type.
316 type: 'String',
317
318 // If not specified then the column is not sortable.
319 sortable: true,
320
321 // Used to compute the name of the RequireJS module that will be used to display/edit the values from this column.
322 // If not specified then the default 'text' displayer is used.
323 displayer: 'link',
324
325 // This is used only by the 'link' displayer (which receives the row data and the column descriptor).
326 linkType: '...',
327
328 // Used to compute the name of the RequireJS module that will be used to filter the values from this column.
329 // If this is not set or set to false then the column is not filterable.
330 filter: 'text',
331
332 // This is used only by the 'text' filter (which receives the column descriptor).
333 match: 'prefix',
334
335 // Optional CSS class name to add to the TH element.
336 styleName: '...'
337 }, {
338 id: 'birthdate',
339 name: 'Birthdate',
340 icon: '...',
341 sortable: true,
342 displayer: 'date-timeago',
343 filter: 'date-range'
344 }, {
345 ...
346 }, {
347 id: '_actions',
348 name: 'Actions',
349
350 displayer: 'actions',
351 // This is used only by the 'actions' displayer (which receives the row data and the column descriptor).
352 actions: [
353 {id: 'delete', label: 'Delete', icon: '...'},
354 ...
355 ]
356 }
357 ],
358
359 // The list of columns to display.
360 columns: ['doc.title', 'birthdate', ...],
361
362 // The list of columns to sort on. The sort order can be optionally specified.
363 sort: ['birthdate:desc'],
364
365 // The values used to filter the live table rows. The live table filters are prefilled with these values. When a new
366 // filter is applied this configuration is updated.
367 filterValues: {
368 birthdate: ['...', ...],
369 ...
370 },
371
372 //
373 // Read-only
374 //
375
376 // The list of known column types. If this is specified then allow the user to choose the column type when creating a
377 // column (and prefill the column descriptor with the settings from the column type).
378 columnTypes: [
379 {id: 'String', name: 'String', icon: '...', sortable: true, displayer: 'text', filter: 'text'},
380 {id: 'Date', name: 'Date', icon: '...', sortable: true, displayer: 'date', filter: 'date-range'},
381 ...
382 ],
383
384 // The list of known filters to choose from when creating a new column.
385 filters: ['text', 'date-range', 'user', ...],
386
387 // The list of known displayers to choose from when creating a new column
388 displayers: ['text', 'html', 'link', 'date', 'date-timeago', 'user', ...],
389
390 // Limit the number of rows displayed.
391 limit: 10,
392
393 // The maximum number of page links to display in the pagination.
394 maxPages: 10,
395
396 // Used to build the page size drop down.
397 pageSizeBounds: {min: 10, max: 100, step: 10},
398
399 // This is used to compute the name of the RequireJS module that will be used to get the live table data. If not
400 // specified, a default source is used.
401 source: '...',
402
403 // This is used by the live table source module to get the live table data.
404 url: '...'
405 }
406 {{/code}}
407
408 If we want to make this generic in order to handle any data representation (table, card, calendar, etc.) then the configuration could looks like this:
409
410 {{code language="js"}}
411 {
412 //
413 // The query
414 //
415
416 query: {
417 // The list of properties to fetch.
418 properties: ['title', 'year', ...],
419
420 source: {
421 // This is used to compute the name of the RequireJS module that will be used to get the data. If not specified, a
422 // default source is used.
423 id: '...',
424
425 // This is a parameter used by the specified source module.
426 url: '...'
427 },
428
429 // The constraints to apply on the property values. These are hidden constraints that the user cannot change.
430 hiddenFilters: {
431 year: ['...', ...],
432 ...
433 },
434
435 // The visible filter values that the user can change. The filters are prefilled with these values. When fetching
436 // the data this is merged with the hidden filters (see above).
437 filters: {
438 year: ['...', ...],
439 ...
440 },
441
442 // The list of properties to sort on. The sort order can be optionally specified.
443 sort: [{
444 property: 'birthdate',
445 descending: false
446 }, ...],
447
448 // Indicates where the current page starts.
449 offset: 0,
450
451 // The number of entries to fetch (the page size).
452 limit: 10
453 },
454
455 //
456 // The data
457 //
458
459 data: {
460 // The total number of entries available (on the server side).
461 count: 54,
462
463 entries: [
464 {
465 // property: value
466 title: 'Work from home',
467 year: 2020,
468 ...
469 },
470 ...
471 ],
472 },
473
474 //
475 // The meta data (used to control how we interact with the data)
476 //
477
478 meta: {
479 // Describes the properties that may appear in the data set. This determines the list of known (available)
480 // properties. Creating new properties, removing existing properties as well as editing the property descriptor
481 // should be done through this array.
482 propertyDescriptors: [
483 {
484 // Identifies the property that this descriptor corresponds to.
485 id: 'title',
486
487 // The property name. Could be displayed before the property value.
488 name: 'Title',
489
490 // Could be displayed when hovering the property name.
491 description: '...',
492
493 // Could be displayed before the property name, if specified.
494 icon: '...',
495
496 // The property type, selected when creating the property. It is used to prefill the property descriptor.
497 // Could be mapped to an xclass property type.
498 type: 'string',
499
500 // Whether the user can sort on this property or not. If not specified then the property is not sortable.
501 sortable: true,
502
503 // Displayer configuration.
504 displayer: {
505 // Used to compute the name of the RequireJS module that will be used to display/edit the values from this
506 // property. If not specified then the default 'text' displayer is used.
507 id: 'link',
508
509 // This is used only by the 'link' displayer (which receives the data entry and the property descriptor).
510 linkType: '...'
511 },
512
513 // Filter configuration.
514 filter: {
515 // Used to compute the name of the RequireJS module that will be used to filter the values from this property.
516 // If this is not set or set to false then the property is not filterable.
517 id: 'text',
518
519 // This is used only by the 'text' filter (which receives the property descriptor).
520 match: 'prefix'
521 },
522
523 // Optional CSS class name to add to the HTML element used to display this property.
524 styleName: '...'
525 }
526 ],
527
528 // The list of known property types. When creating a new property the user can select from this list and the
529 // property descriptor will be prefilled based on the selected property type.
530 propertyTypes: [
531 {id: 'string', name: 'String', icon: '...', sortable: true, displayer: {...}, filter: {...}},
532 ...
533 ],
534
535 // The list of known filters to choose from when editing the property descriptor.
536 filters: [
537 {id: 'text', ...},
538 {id: 'date-range', ...},
539 {id: 'suggest', ...},
540 ...
541 ],
542
543 // The list of known property displayers to choose from when editing the property descriptor.
544 displayers: [
545 {id: 'text', ...},
546 {id: 'html', ...},
547 {id: 'link', ...},
548 {id: 'actions', ...},
549 ...
550 ],
551
552 // Configure the pagination display.
553 pagination: {
554 // The maximum number of page links to display in the pagination.
555 maxShownPages: 10,
556
557 // Used to build the page size drop down that allows the user to change the number of entries displayed per page.
558 pageSizeBounds: {min: 10, max: 100, step: 10}
559 }
560 }
561 }
562 {{/code}}
563
564 The live table widget needs to trigger some events. See the current list of [[live table events>>xwiki:Documentation.DevGuide.FrontendResources.JavaScriptAPI.WebHome||anchor="HLivetableevents28livetable.js29"]] for details.
565
566 * when the live table is ready (initialized)
567 * whenever the live table rows are updated (due to pagination, sorting, filtering, etc.)
568 * after a row is displayed
569 * when a column is created, added, removed, edited or deleted
570 * when a row is created
571
572 The live table widget is currently implemented by [[livetable.js>>https://github.com/xwiki/xwiki-platform/blob/xwiki-platform-12.3/xwiki-platform-core/xwiki-platform-web/src/main/webapp/resources/js/xwiki/table/livetable.js]].
573
574 == Filters ==
575
576 The filter JavaScript modules help the user pick the live table filter values. There are 2 types of (visual) live table filters:
577
578 * column filters: displayed below the column header, they filter the values from that column
579 * external filters: displayed outside the live table (e.g. before), they are not bound to a column and they filter the live table rows based on values that are not necessarily displayed
580
581 We need at least the following pickers for column filters:
582
583 * [[date range>>https://github.com/xwiki/xwiki-platform/blob/xwiki-platform-12.3/xwiki-platform-core/xwiki-platform-web/src/main/webapp/resources/js/xwiki/table/livetabledate.js]]: allow the user to select a date range using for instance 2 calendars
584 * [[list with multiple selection>>https://github.com/xwiki/xwiki-platform/blob/xwiki-platform-12.3/xwiki-platform-core/xwiki-platform-web/src/main/webapp/resources/js/xwiki/table/livetablemulti.js]]: allow the user to filter the column by selecting one or multiple values from a limited list of known values
585 * [[suggest picker>>https://github.com/xwiki/xwiki-platform/blob/xwiki-platform-12.3/xwiki-platform-core/xwiki-platform-web/src/main/webapp/resources/uicomponents/suggest/suggestPropertyValues.js]]: help the user select a value from an unlimited list of values by suggesting the existing values
586
587 We need to support at least the following external filters:
588
589 * tag cloud: filters the live table rows based on the tags associated to wiki pages (when the live table rows represent wiki pages)
590
591 The live table widget shouldn't be aware of the filter pickers and external filters. The filter picker modules are included by the Velocity macro based on the live table configuration (e.g. the date range picker is included only if there is a date column) and they enhance the live table filters using the events triggered by the live table widget.
592
593 == Displayers ==
594
595 The displayer modules are responsible for displaying each cell, both when viewing and editing the live table data. They could be implemented as RequireJS modules and the live table configuration would specify which displayer module to use for each column.
596
597 {{code language="js"}}
598 define('xwiki-livetable-displayer-date', [...], function(...) {
599 return {
600 view: function(column, rowData, tableConfig) {
601 ...
602 return domElement;
603 },
604 edit: function(column, rowData, tableConfig) {
605 ...
606 return domElement;
607 }
608 }
609 });
610 {{/code}}
611
612 The following displayers are needed:
613
614 * **default**: simply display the value for view and a plain text input for edit
615 * **date**: support time-ago format and a date picker for edit
616 * **page**: link to page on view and page picker when editing
617 * **user**: user avatar and link to user profile on view and user picker on edit
618
619 == Synchronization ==
620
621 This module is responsible for updating the live table when the data is modified on the server. There are two types of synchronization that we can achieve:
622
623 * **Closed synchronization** (less complex)
624 ** Doesn't depend on the data source
625 ** Only the changes done within the real-time session are propagated (e.g. only the changes done through the live table)
626 * **Open synchronization** (more complex)
627 ** Depends on the data source (specific change listeners have to be written for each data source)
628 ** Any change done to the data source is propagated (e.g. the live table is updated when one of the pages that is listed is modified outside of the live table)
629
630 For the scope of this proposal we will focus only on **closed synchronization**.
631
632 = User Cases =
633
634 The brand new feature of the Livetable 2.0 is real-time editing.
635
636 The entries can be directly edited from the Livetable fields. The modified objects are updated on the server, and in all the other instances of the Livetables displaying this data, without needed to reload the page.
637
638 If two user are editing an entry at the same time, existing conflicts should be resolved automatically.
639
640 User Case 1:
641
642 (% class="box" %)
643 (((
644 //user1// and //user2// are viewing the results of the same Livetable
645
646 * //user1// modify a property of an object directly from the Livetable
647 * the object is modified on the server
648 * //user2// gets its own Livetable instantly updated
649 )))
650
651 Similarly, if the entry is modified from its edit page, it has to update the opened Livetables too.
652
653 User Case 2:
654
655 (% class="box" %)
656 (((
657 //user1// is browsing a Livetable, while user2 is currently editing an object in its edit page
658
659 * //user1// modify a property in the same object directly from the Livetable
660 * the object is modified on the server
661 * //user2// do not get the update about the modification, as the page is static
662 * //user2// modify a property in the object and save the changes
663 * the object is modified on the server
664 * //user1// gets its Livetable instantly updated, and his modification gets overridden
665 )))
666
667 Moreover, an object can be displayed in Livetables other than the one of its application. Any Livetable using an object should watch for any updates of this object.
668
669 User Case 3:
670
671 (% class="box" %)
672 (((
673 //user1//, user2 and //user3// are viewing the results of different Livetables
674
675 * //user1// modify a property of an object directly from the Livetable
676 * the object is modified on the server
677 * //user2// have this object on its Livetable, so it gets instantly updated
678 * //user3// does not have this object on its Livetable, so nothing changed for him
679 )))
680
681 The User Case 3 is more complicated to set up, because we have to watch for the update of any entries of the Livetable, and no longer of the Livetable itself.
682
683 == Beyond the real-time Livetable ==
684
685 This project is the first step of creating a real-time editable wiki.
686
687 The end-goal would be to update in real-time not only the Livetables, but also any occurrence of an object in the wiki.
688
689 User Case 4:
690
691 (% class="box" %)
692 (((
693 //user1//, is viewing the results a Livetable, and //user2// is browsing some wiki pages
694
695 * //user1// modify a property of an object directly from the Livetable
696 * the object is modified on the server
697 * //user2// have that property of the object displayed on its page, and it gets updated
698 )))
699
700 This is not part of the project, but this could be taken in account, to built code that could extend to this purpose in the future.
701
702 = The Livetable Implementation =
703
704 == Logic vs Layout script ==
705
706 A good feature of the new Livetable would be to support different layouts for displaying the data to the user. The table layout is obviously the first one we need to implement, but we could also add other ones like cards layout or calendar layout when the first one is done.
707
708 If we want to implement such a feature, we need to think our code in two separated layers:
709
710 * the layout script
711 ** only display what it receives from the logic script
712 ** listen to user interaction, and notify the logic script appropriately (e.g. the user click on the first column header, it tells the logic part to sort the column 1 in reverse direction)
713 ** there can be different layout scripts: table, cards, calendar, timeline, ...
714 * the logic script
715 ** handle all the logical operations: filtering, sorting, pagination, URL hash, ...
716 ** intermediate between the macro and the layout script
717 ** when it received a change notification from the layout script, it apply the changes and fetch the new data from the server accordingly, then pass the new data to the display layout
718 ** works like an API, can communicate with other XWiki widgets, is easily extendable
719
720 As the user may want to switch between the Livetable available layouts, the logic script could dynamically import the layout scripts it needs. The available layouts should be specified in the macros.
721
722 == Editing mode vs Designing mode ==
723
724 The Livetable should be able to handle two modes:
725
726
727 === Edit mode (local) ===
728
729 The user can modify the data in existing rows and add new rows.
730
731 The configuration modifications are local, and can be saved with the URL hash. This includes:
732
733 * switching layout
734 * filtering, sorting, columns visibility
735 * other layout specific configuration
736
737 The user cannot perform actions locally on the data structure, like adding properties locally (this would make no sense).
738
739 === Design mode (global) ===
740
741 The user can still modify the data.
742
743 The configuration modifications are shared between users in design mode, but are not pushed to the server directly. It allows users in edit mode to not receive the modifications, and be able to locally view / edit the data without being bothered by the structured being changed at the same time.
744
745 The layout configuration are global, and will be saved as default configuration. This includes:
746
747 * choosing default layout
748 * choosing default filters, sort, columns visibility (and )other layout specific configuration
749 * adding / deleted properties
750 * changing type of existing properties
751
752 Users with the appropriate rights can push the new Livetable config to the server. This will update the Livetable struture in edit mode too (or show a notification asking to refresh the Livetable in oreder to see the new structure, so that users do not lose their current local config right away).
753
754 == The table layout technologies ==
755
756 For each layout, we might need to use a library to help us displaying the data.
757
758 In this proposal, we will focus only on the table layout, as it was only the first one required.
759
760 === Solution 1: Using a table library ===
761
762 We could use a JavaScript library to create HTML tables from existing data. This would simplify and speed up the development of the new Livetable.
763
764 However, the library will probably not handle the user interactions around the intended features (sorting, filtering, modifying...) the way we want. We will have to rewrite those functionalities in the way we want them to behave. This might become a problem if there are too many functionalities we have to rewrite. We will end up with a whole library where we only use the displaying part of it, and a lot of functions overriding the rest of the existing functionalities.
765
766 What we could do to solve these problems is to create a facade / adapter object that uses the library functionalities, and adapt them to our needs. Also, in this way the implementation will stay rather independent from the chosen library, and we will be able to upgrade or change the library in the future by just updating the adapter code. This would be great for maintainability in the future years
767
768 Several libraries could serve this purpose:
769
770 ==== [[Tabulator.js>>http://www.tabulator.info/]] ====
771
772 pros:
773
774 * development very active (new release every 2~~3 months)
775 * modular implementation: we can easily overwrite and adapt the existing functionalities
776 * already comes with the intended features: display / editing cells by column type. We can add custom formatters / editores for the types we want.
777
778 cons:
779
780 * does not natively support IE11, polyfills needed
781 * not very responsive on small layout (can only hide or collapse columns, but collapsed columns cannot be editable through Tabulator native edit system)
782 * didn't find how to edit cells with dblclick instead of single click
783 * native column reorder is a bit buggy
784
785 ==== [[Datatables>>https://www.datatables.net/]] ====
786
787 pros:
788
789 * development still active
790 * lightweight and configurable
791 * very easy to add custom formatters for columns
792
793 cons:
794
795 * does not support editable cells by default
796 * not really responsive (has an extension to collapse columns but it looks awful, however we could create our own responsive extension)
797 * column formatters work with outerHTML and not HTML elements, so they would be more limited
798 * if a property does not exists in an entry (i.e the property is not empty but undefined), it window.alert an error by default (we can override that but... why?)
799
800 Others solutions are available, but they are less configurable, no longer in development, or need a commercial license.
801
802 === Solution 2: Using a framework ===
803
804 We could also re-write entirely the Livetable component by ourselves. As opposed to the solution 1, this would imply a little more development time, but greater flexibility and maintainability. Instead of using existing code that we have to adapt, we would already build code that meet our needs in the first place. Furthermore, if the layout script only need to display data and handle user interaction, it might not be too complex to create a table from scratch.
805
806 In this case, we should use a framework to fasten the development, by automatically binding the HTML table to the JSON object, and helping with event handling.
807
808 [[Vue.js>>https://vuejs.org/]] would be a great solution, it is versatile and easy to set up. With this framework we would be able separate our Livetable in smaller components that could communicate with each other. We could think of components like table rows, table headers, or column filters. With this technique, the code would become more understandable and maintainable in the future.
809
810 We will have to add an event adapter for Vuejs so that it can communicate with the rest of the page.
811
812 We can also use some utility libraries to help us during the development, like libraries for drag-and-drop, custom selects, or menus.
813
814 == Implementing existing features ==
815
816 The features of the first version should still be present in the new one.
817
818 These features include:
819
820 * Filtering entries
821 * Sorting entries
822 * Pagination
823 * Actions to view, edit or remove the entries
824 * Responsiveness
825
826 These features has to be implemented differently from the first version, to allow more flexibility.
827
828 For example, the Livetable of [[Notion.so>>https://www.notion.so]] allows users to perform complex filtering and multi-columns sorting. However, theses features are not always displayed to the user, and are hidden under a sub-menu of each feature respectively.
829
830 It would be nice to find a compromise between the complexity of the feature and its easiness to use for the user, and to have the options visible right away.
831
832 Notion also allows the user to modify not only the data but also the structure of the table (adding / removing columns, changing type of column, ...). When displaying a Livetable on a page this is not desired, the user should only be able to modify permanently the data. What we could do is let the user perform several operations on the columns only on its local instance of the Livetable, like re-ordering, showing or hiding them.
833
834 As several display layout would be available (table, cards, ...), we also need to consider if we want these features to be implemented in the same way for each layout. If it's the case, that means less development time, and more coherence between the layouts. However, certain layout could have a better suited implementation that would be more intuitive for the user.
835
836 === Sorting ===
837
838 For the moment, we can only sort the table according to one column.
839
840 Implementing a multi-sort intuitive for the user is quite complicated. Most of existing sortable tables does not implement multi-sorting.
841
842 For those implemented such a system, they came with different implementations:
843
844 * keeping the previous sorts order for sub-level sorting (Google Sheets)
845 ** pros: really simple, no wizard or sub-menu needed
846 ** cons: not practical at all for the user experience
847 * using a wizard (Libre Office): we can choose which column correspond to which sort level
848 ** pros: easy to understand and perform
849 ** cons: not easily accessible, not easily readable (the user cannot see with one glance how to column are sorting the table)
850 * using a sub-menu (Notion.so): we access in a sub-menu a sortable list of the columns used for the multi-sort
851 ** pros: easy to perform, a bit more accessible than using a wizard
852 ** cons: still not accessible / readable right away
853
854 Each of the described solutions makes compromises, and come with pros and cons.
855
856 Moreover, in most of the cases, we don't have any indication about which column correspond to which level of sorting.
857
858 In our final solution, we will need to get the sort-system as accessible and readable as possible.
859
860 ==== Solution 1 ====
861
862 This is our custom solution for multi-sorting column.
863
864 We can add a number next to the triangle indicating the level of sorting:
865
866 [[image:image-20200511112549-12.png||height="88" width="560"]]
867
868 (Or with color to be more distinguishable from each other)
869
870 [[image:image-20200511160945-2.png||height="89" width="561"]]
871
872 Not all the columns have to sort the data, here only //Title// and //Date// are used (unused columns shows a ghost triangle on hover):
873
874 When we left-click on the icon (or the whole title):
875
876 * if the column was already sorting (at any level), it changes the direction of the sort (asc / desc)
877 * else, it became the new level-1 sort
878
879 When we right-click on the icon, a sub-menu appears to let us choose the level of sort we want:
880
881 [[image:image-20200511161346-4.png||height="106" width="565"]]
882
883 * if we click on the current sort level, it changes its direction
884 * else, it overrides the existing sort of the chosen level
885
886 Only the useful levels are displayed in the sub-menu: an already sorting column will not show lower level sorts, and a non-sorting column will show levels until the lowest level possible in the current configuration.
887
888 Pros:
889
890 * easy to visualize how the columns are sorted
891 * quick to modify the different sort levels
892
893 Cons:
894
895 * right-clicking might not be intuitive for the user, and he might never know it's possible
896 * the colors of the sort levels, even though they are common, might not looks good in some themes, or might not be wanted by the user (solution: every level stays black)
897
898 ==== Solution 2 ====
899
900 This second solution is a variation of the first one. Instead of displaying the direction and the level together, we can display both information separately:
901
902 [[image:image-20200515112513-1.png||height="87" width="572"]]
903
904 When we hover a title with no sort on it:
905
906 [[image:image-20200515112904-2.png||height="82" width="300"]]
907
908 When we click on the title or the triangle icon: same behavior than the left-click of solution 1
909
910 When we click on the "+" button: same behavior than the right-click of solution 1
911
912 Pros:
913
914 * same pros than solution 1
915 * no more unintuitive right click
916 * as the level is displayed on its own, we no longer need to impose the colors to ease the distinction
917
918 Cons:
919
920 * it might not be clear that the plus sign refers to creating a new level of sort
921
922 (Note: with this solution we could still keep the behavior of the right-click as an alternative way to open the sort level sub-menu.)
923
924 ==== Solution 3 ====
925
926 We create a sub-menu above the table where we can specify the order of the columns in the multi-sort (like in Notion). This solution would work on any layout.
927
928 Pros:
929
930 * keeps the UI light
931 * works on any layout
932
933 Cons:
934
935 * functionality hidden behind a menu: we are adding extra steps to sort the table
936 * we can't know with a glance how the columns are sorted
937
938 We could still display the triangle icon in the table header to indicate that the column is currently sorting the table. Clicking on it would open the sort menu.
939
940 === Filtering ===
941
942 For the moment, the filter only try to see if the rows match the specified text. It is not possible to check for the inequality of a number, nor for the range of a date for example.
943
944 It would be nice to have more complex filtering options, like: "=", "≠", "≤", "≥", "between", ... Specific operators should be added according to the property type (date, list, ...).
945
946 Also, for now it's not possible to combine filters in another way than a "AND" logical operator between the columns. A solution could be implemented to be able to combine different columns with a "OR" operator (e.g. name="jean" OR age≥25), or to be able to combine different filters in the same column (e.g. age≤20 OR age≥50) The combination of "AND" and "OR" operators should be kept intuitive though.
947
948 ==== Solution 1 ====
949
950 For now, we can only have 1 filter for each column, and we can only match equality.
951
952 What we can do first is to append the operator before the input, with the ability to change it by clicking on it:
953
954 [[image:image-20200511114754-15.png||height="195" width="582"]]
955
956 We can also allow multiple filters by adding an action to create more :
957
958 [[image:image-20200513100959-1.png||height="200" width="583"]]
959
960 Clicking "add filter" will automatically focus the created input.
961
962 When there is no filter, there is only the "add filter" action, so that the UI stays light.
963
964 From the second filter, a new button is added allowing to choose between the "AND" and the "OR" operators. The logic operator has to stay the same for all filters in the same column. For instance if "AND" is chosen, all the filters of the column will be combined using the "AND" operator.
965
966 With this current design, there is no other way to change the way columns are combined together. For now, all the columns are combined with the "AND" operator. In any way, the operator should also stay the same between each columns to keep the behavior understandable.
967
968 Here is a test of the design on large layout with a lot of filters:
969
970 [[image:image-20200513104407-1.png||height="189" width="788"]]
971
972 and on narrow layout:
973
974 [[image:image-20200513104531-2.png||height="194" width="433"]]
975
976 On the narrow layout, the fields are becoming too small, and the result becomes quite heavy (we can work on the design of the inputs to clean the design a bit more).
977
978 In addition to that, the filters take much more width than the column content, this is really showing up in number columns like the "Age" one for instance. This might become a problem for large Livetable with a lot of columns.
979
980 However, on average the user is not going to create 3 filters for each columns, and not filters every columns. As the UI complexity is proportional to the filtering complexity, the UI should stay light most of the time.
981
982 Pros:
983
984 * unlimited number of filters for each column
985 * filters directly accessible: the user can quickly filter, and knows instantly what's being displayed
986 * straightforward
987
988 Cons:
989
990 * we can only use the "AND" operator between columns with this current design.
991 * the filters can take much more width than the column content
992 * not lightweight at all
993
994 ==== Solution 2 ====
995
996 We create a sub-menu above the table where we can specify the filters we want to apply to each columns (as Notion does). This solution would work on any layout.
997
998 Pros:
999
1000 * unlimited number of filters for each column
1001 * as opposed to the solution 1, the width of the filters is not a problem anymore
1002 * keeps the UI light
1003 * works on any layout
1004
1005 Cons:
1006
1007 * filters not accessible directly: we are adding an extra step to access the filters
1008 * filters not viewable by default: we can't know with a glance how the columns are filtered
1009
1010 ==== Filtering by Tags ====
1011
1012 There is currently a way to filter by tags, from the outside of the Livetable. As the script layer should behave like an API, it should be easy to communicate with it.
1013
1014 ==== Global search ====
1015
1016 We can also add a global search input above the Livetable, that is going to search if the input text can be found anywhere in the entries of the data source.
1017
1018 Any entries containing the query text in one of their properties will be returned. This filtering system should combine with the normal one, with an "AND" operator.
1019
1020 ==== Filtering operators ====
1021
1022 Filtering a string is not the same that filtering a number, a date, or a list of values. Thus, we should not provide the same actions for all type of filters, but adapt according to the filter type.
1023
1024 When adding a filter on a column, the program has to do 2 things:
1025
1026 * create the operator select, corresponding to the column type
1027 * create the input along with its picker, corresponding to the column type, and the selected operator if needed
1028
1029 There can be different pickers for a same property type. For instance, a "Number" type can be associated to the default number input, or to a custom rating select showing stars. Even if the rating select displays stars to the user, it will return a integer (between 1 and 5) at the end.
1030
1031 Ideally, the operators select should not be aware of anything but the property type. However, as the data source may not come with code written for certain operators for the query creation, and as there is no fallback to a default query creation based on the type of the property, we cannot assume that.
1032
1033 Instead, what we have to do is to disable the operators that are not supported by the data source, indicated in the Livetable JSON configuration object.
1034
1035 For each property type, the default operators should be:
1036
1037 * Text
1038 ** "Equals"
1039 ** "Dos not Equals"
1040 ** "Starts with"
1041 ** "Ends with"
1042 ** "Contains" (default)
1043 ** "Does not Contains"
1044 ** "Before"
1045 ** "After"
1046 ** "Like": allows to perform regex-like filtering for advanced users
1047 ** "Is empty"
1048 ** "Is not empty"
1049 * Number
1050 ** "=" (default), "≠", "<", "≤", ">", "≥"
1051 ** "between": shorthand for "≥ filter1 AND ≤ filter2", displays two inputs instead of one
1052 * Date
1053 ** "Is" / "=" (default)
1054 ** "Is not" / "≠"
1055 ** "Before" / "≤"
1056 ** "After" / "≥"
1057 ** "between": shorthand for "After filter1 AND Before filter2", displays two inputs instead of one
1058 ** "Is empty"
1059 ** "Is not empty"
1060 * List (unique values)
1061 ** "Is" (default)
1062 ** "Is not"
1063 ** "Value In": shorthand for "Is filter OR Is filter OR ...", displays a select
1064 ** "Value not in": shorthand for "Is not filter OR Is not filter OR ...", displays a select
1065 ** "Is empty"
1066 ** "Is not empty"
1067 * List (multiple values)
1068 ** "Contains" (default)
1069 ** "Does not contains"
1070 ** "Values In": shorthand for "Contains filter OR Contains filter OR ...", displays a select
1071 ** "Values Not in": shorthand for "Does not contains filter OR Does not contains filter OR ...", displays a select
1072 ** "Is empty"
1073 ** "Is not empty"
1074
1075 Some notes:
1076
1077 * the filters are case aware: lowercase matches both lowercase and uppercase, while uppercase only matches uppercase
1078 * the operators written in words can take up too much space depending the chosen implementation. This would be the case in the Solution 1.
1079
1080 === Pagination ===
1081
1082 The pagination is already functional and nothing but its design might be changed.
1083
1084 We could also implement an infinite scroll feature that the user could choose instead of the pagination system in the table options. However, this would not be great for every cases, as there might be some content under the Livetable that could not be accessed anymore as the table is continuously expanding. The scroll feature could be done inside the Livetable to fix this issue.
1085
1086 Similarly, we could display a "load more" button at the bottom of the Livetable so that the user manually fetches new entries.
1087
1088 === Data export ===
1089
1090 In the new design, we need to keep the feature allowing the user to export the content of the Livetable to csv format.
1091
1092 This is not the priority, be we could add other formats, like excel or json.
1093
1094 == Implementing new features ==
1095
1096 === Editing cells ===
1097
1098 If the user single-click on a cell, the outline of the cell changes color, showing that the cell is currently focused. A tooltip could be displayed on hover telling to double-click in order to edit. The current focused cell is only visible by the user and is not shared with other viewers of the Livetable.
1099
1100 If the user double-click on a cell, the cell will go into edit mode. The displayer module will be replaced by its corresponding editor module to allow the user updating the value. When the user goes out of focus or the user press "Enter", the changes are kept, and the displayer is used again to display the new value. If the user press "Escape" instead, the cell goes back to it initial value. If the user presses "Tab" while editing a cell, the changes are kept and the next cell is focused in edit mode.
1101
1102 === Adding a new row ===
1103
1104 The user should be able to easily insert a new row in the table.
1105
1106 However, as the Livetable does not display all the properties of the pages, we cannot just insert a new row with all its cells editable, as the created page would be incomplete.
1107
1108 We can create a button above the Livetable to insert a new row in the Livetable.
1109
1110 When the user click on it, the action that follows could be either of:
1111
1112 ==== Solution 1 ====
1113
1114 We simply redirect the user toward the add-new-entry page.
1115
1116 Pros:
1117
1118 * cannot be simpler
1119
1120 Cons:
1121
1122 * we leave the Livetable page to create a new entry
1123 * there are now two buttons with the same effect on the page
1124
1125 ==== Solution 2 ====
1126
1127 We display a wizard in a modal allowing the user to fill all the properties of the new entry. This solution is implemented in Notion.
1128
1129 Pros:
1130
1131 * we do not leave the page
1132
1133 Cons:
1134
1135 * way more complicated than solution 1
1136 * we have to create another wizard doing the same thing than the one in the add-new-entry page
1137
1138 ==== Solution 3 ====
1139
1140 We add a new empty row at the bottom of the Livetable, only if all the properties are displayed to the user.
1141
1142 Pros:
1143
1144 * really intuitive to add new data for the user (coherent with the editing system)
1145
1146 Cons:
1147
1148 * if all properties are not displayed, we cannot use this system
1149 * and using two different systems to add new rows could be confusing for the user
1150
1151 ==== Handling rows in the wrong page ====
1152
1153 When inserting a new row or modifying an existing one, there is a possibility that the row has to be displayed in a different page that the current one, because of the Livetable configuration (sorting, filtering, ...). When this happens, we cannot just move the row to the new page it belongs, as it would be confusing for the user (the row would just disappear with no indication).
1154
1155 To avoid that, we can hold the modified row in the current page with a different state, shown by a different display style. In front of the row, we can display an icon representing an "i" in a circle (for "information"), that explains on hover the current state of the row, and that it will be back to normal on the next user action (change sort, filters, modify another row...).
1156
1157 === Selecting rows and Batch operations ===
1158
1159 This feature has been asked by many users: it would be nice to be able to select rows in the table, and to execute an action for all of them.
1160
1161 The action could be: deleting the entry, changing the rights, modifying value...
1162
1163 This would not be complicated to set up: we only need to add column on the left of the table where we display a checkbox for each row. In the column header, there is a checkbox that allow to quickly check / uncheck all the present rows in the table.
1164
1165 Once we select at least one row, a menu appears (or become enabled) above the table, and let the user perform the desired action.
1166
1167 === Columns operations ===
1168
1169 There are several operations on the columns the user might consider, like reordering or hiding some of them. These action would only affect the user display and would have no effect on the server or the other Livetables.
1170
1171 We can include these changes in the URL hash (automatically or by user action) so that the user can keep the current state of his table, or share it with other people.
1172
1173 Moreover, we can think of a system that allow the user to create a new page based on the current Livetable configuration.
1174
1175 ==== Showing / Hiding columns ====
1176
1177 We can create a sub-menu above the table, showing all the columns the Livetable can display, each column associated with a checkbox allowing the user to toggle its visibility.
1178
1179 If some columns are hidden in the Livetable, we should display an icon in the sub-menu title to indicate that not all the columns are shown. The icon could be an exclamation mark or an eye (or both).
1180
1181 ==== Reordering ====
1182
1183 The easiest way for the user to reorder columns is by drag and dropping them at the place they want.
1184
1185 We could also allow the columns listed in the sub-menu above the table to be order-able.
1186
1187 ==== Resizing the columns ====
1188
1189 This is not the priority but if anyone wants this feature, it could be implemented to the Livetable.
1190
1191 == Other concerns ==
1192
1193 * The new Livetable should be compatible and adaptable with the current styling configuration of the wiki.
1194 * It should also respect the accessibility specifications:
1195 ** be a valid ARIA table
1196 ** be usable with the keyboard only (through tabindex)
1197 * The data received from the server should be cached to alleviate the workload of the server when going back to a previously browsed Livetable page.

Get Connected