Description

At the moment, we filter rights after a database or Solr query. This has the problem that the number of results might be lower than expected, leading to nasty holes or Live Data tables full of N/A values. Further, the counts in facets and the total number of results doesn't match what the user expects. The idea of this proposal is to fix this while accepting some performance impact. This could be optional.

Main Idea

The main of this proposal is to store in the database and in Solr for every document a list of users who cannot access that document. These lists would be used at query time to filter out any document that the current user cannot view. This alone won't work as it will be too costly to fill these lists. However, if we keep the right check after the query, we can afford to treat these lists as an incomplete cache: Whenever a right check in a query fails, we record those failures in the database/in Solr and repeat the query until no right check fails anymore. While this will significantly slow down some queries, the slow-down is only once for every document, and it doesn't repeat even if the concrete search query is different. In order to keep the cache up-to-date, any changes to rights needs to remove all affected denials, similar to the security cache.

From the user experience point of view, in paginated queries, the list of results will always be full and won't contain any holes. However, the total number of results will change as the user views more pages, and it is possible that the user opens a page, and it won't have any results. This could be particularly problematic when the user explicitly jumps to a page. Similarly, the numbers in facets in Solr search will go down as the user searches more/views more result pages.

Implementation Ideas

In the database, the list storage would be a table of user and document ID (integer ID) that has indices for both combinations of user/document (user/document first and the other one second). In Solr, the only option seems to be to store a list property of denied users for every document. Solr supports atomic updates that allow adding a value to a list field if it doesn't exist yet which seems perfect for this purpose. Similarly, atomic updates allow removing values from multivalued fields without having to perform a full update of the document and without any need to synchronize these updates on the XWiki side. In Solr, it isn't easily possible to add data to the indexed documents. Atomic updates are only possible when all fields are either stored or docValues, which is not the case in XWiki's search core. We could store those lists of denied users in a separate Solr core and join them in, but this scales linearly in the number of documents as illustrated in this blog article about Solr and joins. It would also be possible to include the documents to exclude in the Solr query, but this seems prohibitively expensive in large wikis where some users can only access a fraction of the documents. In any case, we should carefully examine how this affects the performance and what we can do to improve it.

A challenge will be updating these lists of users after a right change. Every change to the groups of the user will mean that all denials of that user need to be removed. While this is relatively simple in the database, in Solr it isn't clear how to store the data and updating it might be expensive in either case this means updating every document in which the user is mentioned. However, as the worst case is that the user cannot see some document, it seems okay to do this invalidation in an asynchronous background job. We might also consider not simply removing those values but to re-validate them in order to avoid costly queries when the user lists pages again.

For the queries themselves, we need to make sure that we check big-enough batches of pages to not increase the number of queries too much. For example, when the last document in a query cannot be viewed, we shouldn't repeat the query with just that document denied, as it might be that also the following 1k documents cannot be accessed, and we thus perform 1k queries until all of them have the user in the list of denied users. Instead, we should always request some more documents than actually needed so we can ensure that we update documents in big-enough batches and don't need to repeat the query at all if just a few documents have been removed.