Notification status storage

Last modified by Thomas Mortagne on 2020/02/10 14:08

This page is dedicated to gather information and detail the implementation of Solution 2C described on NotificationsOptimization.

The core idea is to move on user configuration based filtering of notifications from read to write step. This have a big impact on the filering and gathering API and architecture but the way to define notifications types should not be impacted.

Filtering

Instead of filters designed to generate SQL queries the new filters should be dynamic filter components which simply indicate directly if a notification should be stored for a specific user(/group ?) or not. This probably means we cannot reuse anything from the current filtering system and instead design a new system from scratch without any "pollution".

Storage

We need to store a potentially high volume of very small entries (not much more than a notification and user ids plus a boolean to indicate a read status and probably a few more metadata coming from the notification like the date depending on the engine). So we need an easy to embbed nosql engine (even better one we already have) which fit well this use case. We also need to think about clustering use case.

The XWiki Hibernate storage

accept We already have it
accept The table already exist
accept No duplication and possibility to join the notification table and the rest of the DB
accept Already covers clustering use case
accept Easy to migrate
cancel Most SQL engines are really not really supposed to be great with this use case and we would need ALL databases engines supported by XWiki to support it well so that's a pretty big constraint

Tests:

100000000 entries in the event status table

  • turns out the perf is actually not bad as long as you setup the right indices. It not the case with the existing table when you are searching the events of a specific user but adding a index on the user reference column made requests very fast

Solr

accept We already embed it
accept Already covers clustering use case
cancel Not exactly optimized for this use case (to be tested)
cancel Not easy to migrate

Lucene

accept We already embed it
accept No need for any setup/schema
cancel No clustering support
cancel Not exactly optimized for this use case (to be tested)
cancel Not easy to migrate

HBase

accept Designed for this kind of use case
accept Remote solution for clustering use cases
cancel We don't have it for other use cases
help Embbedable ?

Not embbedable (for default XWiki Standard)

  • Not in the JVM: Redis, Cassandra, PostgreSQL JSON, ArangoDB, CouchDB
  • Not a compatible license: Neo4j, MongoDB

Gathering

When gathering users notifications there should not be any filtering other than:

  • the user
  • the date
Tags:
Created by Thomas Mortagne on 2020/02/07 13:46
    

Get Connected