Description

While the file system is nice for simple deployments, because it requires no setup, it also has its challenges in terms of backup and when in the context of a cluster (for shared, or shareable, data like attachments and job logs). This is particularly true for deployments in Kubernetes. For this reason, XWiki shouldn't require any permanent file storage.

Goals

• Allow deploying XWiki without anything stored in the local file system that would need to be backed up
• Allow cluster setups without the need to set up any kind of shared file system between cluster nodes
• Support storing (bigger) files like attachments either in the local file system or in S3.
• Allow migrating from local file system storage to S3 and back.

Non-Goals

• Stop supporting the file system. The file system should still be the default storage, S3 shouldn't be a requirement.
• Work without local file system. We still assume that a local file system is available, e.g., to store temporary files, but those files might be cleared between starts of XWiki.
• Handle Solr. We assume that in such setups, an external Solr installation is used and its handling of files is out of scope.
• Store everything that is currently in the file system in S3. As the performance characteristics of S3 are different from the local file system, this won't be possible. Instead, we need to determine what should be stored in the database and what in the file system/object store.

Design Questions

• What directory hierarchy should we use?
  • Keep it the same as we have now? Seems to be the most backwards-compatible option. → This option was implemented.
  • Implement a content-addressable storage, using hash values of the content as identifiers and mapping file names to content in the database?
    • Advantage: reduces storage for attachments
    • Disadvantage: harder to inspect manually and possibly harder to migrate.
• How do we synchronize writes if we don't use content-addressable storage?
  • S3 supports conditional writes that we could use to ensure that we have the expected content before writing, but it's not clear if this would be enough. → Was implemented with conditional writes, but we need to revisit this as it turns out that many alternative S3 implementations don't support them.
  • We could also store and update synchronization information in the database?
  • Writes could be synchronized by the caller for now, see how far we can get with that.
• What do we cache in the local file system, how do we keep the cache up-to-date?
  • Don't cache anything for now.
• How can we support installing extensions in such a setup? Could we support mirroring/caching the full extension repository locally? This probably requires careful design for upgrades.
  • Extensions could be stored in S3 and downloaded/synchronized on instance start or when a node signals that an extension has been installed.  Deferred to https://jira.xwiki.org/browse/XCOMMONS-3251

Design for Individual Use Cases

Job Statuses and Logs

Logs are frequently appended files which isn't a use case that is well-supported by S3. It probably makes more sense to store job logs in the database. This will also make querying and filtering easier. Alternatively/additionally we could also store them in Solr to have better search support. The problem with Solr is that commits are comparatively expensive and might thus limit the immediate availability of job logs.

Job statuses can be large. In principle, we could store them in the database, too, as we fully load them in memory. Alternatively, we can store them in S3 and just store some job metadata in the database to make listing them easier.

We cache job statuses in memory, but it's not clear that this cache is properly invalidated across cluster nodes.

Planned in https://jira.xwiki.org/browse/XCOMMONS-3249 as follows:

• Job logs are stored in the database
• The summary of the job status is stored in the database, the full job status is stored in the blob store (so S3 when activated, filesystem otherwise).
• Via the existing APIs, the job status is only available on the instance where it has been created, so cross-cluster caching is not an issue.

Attachments

We do not seem to require many operations for them, main operations seem to be reading and writing single files.

We should have a cache to avoid constantly re-accessing frequently attachments like the logo. The cache should be size-limited, we probably shouldn't try storing huge files in it. 

Question: how to deal with cases where we read files partially, should we still download and cache the whole file? Or could/should we somehow cache chunks of a file? Answer: Don't cache for now.

Question: when parsing files using Tika or converting files using JODConverter, do we need any kind of random access that might scale badly with S3? Answer: should be okay, they probably already write to files when needed.

Extension Repository

The whole metadata is already loaded in memory. We need to provide a URL targeting the JAR to the classloader, but it could come from anywhere and the classloader itself is going to do caching.

Still, as the extension repository most likely won't change much, we should probably apply extensive caching to the extension repository.

Extension History

Contains XML-serialized extension history statuses. Question: Do we really need XML serialized files? Could we transform this into a database table, maybe only in the main wiki?

Deleted Attachments and Documents

Access should already be streaming, so this should work in general. We should probably use caching to avoid repeated accesses. Note: no caching implemented for the first version, we assume that S3 is fast.

Cluster node ID

This should never be shared between cluster nodes so storing it in S3 makes little sense. Instead, the ID should be provided as environment variable. Still, it might make sense to store it in S3 just for consistency and in case S3 is used in a non-clustering setup. Not handled or planned for now.

Third-Party Libraries

We could use an existing library that provides an abstraction over file system and S3.

Apache Commons VFS

Advantages:

• Supports caching

Disadvantages:

• No built-in S3 support, but only via extension
• At least one of those doesn't make the best impression: https://github.com/abashev/vfs-s3/issues/62 (the other is https://www.leisenfels.com/vfslibdocs3)
• Caching might not really support our clustering needs

Java NIO

Advantages:

• Native Java API, no third-party library
• S3 support is provided by AWS (but as "labs") https://github.com/awslabs/aws-java-nio-spi-for-s3

Disadvantages:

• Some incompatibilities: https://github.com/awslabs/aws-java-nio-spi-for-s3?tab=readme-ov-file#s3-uri-and-java-uri-incompatibility
• Doesn't expose the full S3 API, e.g., cache refresh or conditional writes might simply be impossible
• Caching would still need to differentiate between the two backends as we don't want to cache for the file system

File Names

Naming guidelines of S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html

Object key names are case-sensitive, supporting up to 1024 bytes of UTF-8. This is significantly shorter than the file system, where we typically have no limits on the total length and 255 characters per segment.

They also recommend a subset of safe characters and some characters to avoid. They explicitly recommend avoiding % which we currently use for encoding for job statuses.