OLD: http://markmail.org/message/4ncp4f4sw3fkd2bn
NEW: https://markmail.org/message/kfrtxpu5cyqeykim

 

https://github.com/xwiki-contrib/docker-xwiki
https://dev.xwiki.org/xwiki/bin/view/Community/Testing#HSelenium3-basedFramework

Description

Use Cases

  • UC1: Fast to start XWiki in a given environment/configuration
  • UC2: Solution must be usable both for running the functional tests and for distributing XWiki
  • UC3: Be able to execute tests both on CI and locally on developer's machines
  • UC4: Be able to debug functional tests easily locally
  • UC5: Support at least the following configuration options (i.e we can test with variations of those and different versions): OS, Servlet container, Database, Clustering, Office Server, external SOLR, Browser
  • UC6: Choose a solution that's as fast as possible for functional test executions

Implementation Ideas

In Java Tests

Prepare the full Docker environment from within the Java Test (in JUnit5).

Implementation:

  • Docker containers for the tests are created directly from the JUnit tests. There are two frameworks that can help and meant for this: Jupiter Selenium and TestContainers.
    • We are leaning towards TestContainers which is more generic. Jupiter Selenium focuses only on Selenium containers.
    • Right now TestContainers provides less browser support and doesn't support JUnit5 but we can create our own JUnit5 extension for that.
  • The Servlet container images used will be the official ones (e.g. Tomcat one) and the XWiki WAR will be mapped through docker volumes for performance (so that we don't need to generate a new image for each build)
    • An alternative that is being explored is to the usage of JBoss Undertow to create an embedded Servlet Container in order to make starting XWiki as fast as possible
    • With this solution we won't need to copy the JARs making WEB-INF/lib thus saving build time
    • Also we wouldn't have to rewrite the equivalent of the Packager Maven plugin's package mojo in pure java (no Maven). We would just need to populate the DB as with the "In Maven Build" solution below.
    • Real supported containers such as Tomcat will be tested during the official XWiki Docker image smoke tests
  • The official docker images for DBs will be used (MySQL, Oracle, etc). See https://www.testcontainers.org/usage/database_containers.html
  • Official Docker images are created in the Maven build using the Fabric8 plugin as in the "In Maven Build" proposal.
  • However functional tests don't execute with the official docker images since that would take too long and we want to have minimal XWiki WARs (i.e. that contain only the minimum needed for the tests).
    • The official docker images will have smoke tests in the flavor distribution tests to ensure they work fine.

In Maven Build

The idea of this solution is to implement it at the level of the Maven build.

This idea was dropped since it's nicer to be able to have the whole system set up from the XWiki Test, thus allowing to execute tests directly in your IDE.
  • Use Docker (matches UC1 specifically). Note that UC1 and UC6 removes the VM solution.
  • UC2 implies that the usage of Docker should be done in XWiki's Maven build and not using a Docker plugin for the CI
  • Use the Fabric8 Maven plugin (a.k.a DMP)
  • UC5 means that we need to have a Docker image that contains the various browsers we want to test with (one or several images) and that this image will start the execution of the tests since it's the selenium tests which connect and start the browser.
  • UC2 has the consequence that we need to move the Docker images currently located at https://github.com/xwiki-contrib/docker-xwiki into xwiki-platform

Options available:

  • Option 1:
    • Base image with XWiki
    • One image per supported-DB, containing the data for XWiki Standard
    • Functional tests would all execute on XWiki Standard
  • Option 2:
    • Base image with XWiki
    • Import extensions (JARs + XARs) when using the image, using a Maven Mojo (from the XWiki Packager maven plugin for ex). This mojo would use the XWiki REST API to run Job(s) to import extensions inside XWiki.
    • Only import what's needed for the functional tests
  • Option 3:
    • Base image without XWiki (just the Servlet container)
    • Generate the XWiki WAR using a mojo from the XWiki Packager maven plugin and map a docker container volume to point to this WAR
    • The WAR would contain only what's needed for the functional tests
  • Option 4:
    • Base image with XWiki
    • Generate the XWiki permanent directory and map a docker container to point to it. Generate it with the DataMojo from the XWiki Packager maven plugin. Also populates the DB.
    • Only import what's needed for the functional tests

Pros and cons of options:

  • Option 1 doesn't allow to discover required dependencies for each module and doesn't ensure that a given module works without other modules
  • Option 1 allows testing that all modules work with all the other ones. Note that with the other options we mitigate this with our XWiki Standard functional tests
  • Option 1 is a lot slower for test execution (i.e. failing UC6)
  • Option 2 and 4 are similar. Option 4 seems simpler to use since the mojo already exists
  • Option 3 doesn't really have any advantage and it fails UC2 (it's not the same image used for distribution and tests)

So preferred solution is option 4 ATM.

Usage

  • To run the functional tests for a module: mvn install
  • To generate the XWiki Docker image in that module: mvn install
  • To deploy the XWiki Docker image to Dockerhub: mvn deploy
  • To manually start XWiki in a functional test module: mvn docker:start. Then you can execute the JUnit tests in your IDE for example (however you'll need to have the target browser installed locally)
    • Question: Would it be possible to use the "browser docker image" to not have to have the target browser installed locally?
  • To manually stop XWiki in a functional test module: mvn docker:stop
  • To see the Docker container console logs in a functional test module: mvn docker:logs

Architecture

xwiki-docker.png

Caveats/Problems

  • Right now we publish official XWiki Docker images to https://hub.docker.com/_/xwiki/. However this is not compatible with the strategy of building the XWiki Docker images as part of our build (since the system for official Docker images require to point to a SHA1 from a git repository containing the Dockerfile and ancillary files).
  • Functional tests for the Standard flavor (there are several modules) will take a lot longer to execute unless we find a way to reuse the DB data between the modules.
  • For the ActiveInstalls functional tests we need to have ES started *before* XWiki is started. Right now the ES instance we use is a stub started by the tests and on which we make verifications, so we have a problem.
    • We could start a real ES instance but it would be much more costly and not as easy to verify expectations. 
    • Another solution would be to not use Fabric8's DMP and instead start/stop Docker from Java (for example using docker-client from Spotify - see also manual). Idea: use Fabric8 DMP to create/publish the docker image but use docker-client from Java (from the test code to be precise, in the location where we start XWiki today) to start/stop/wait on containers.
    • However starting XWiki from Java rather than from Maven is less interesting since if you want to perform manual tests on the XWiki instance you need to start the test, instead of just running mvn docker:start for example. 
    • Also, AFAIK, ActiveInstalls is the only functional test that requires to execute something before XWiki is started so it's probably better to find a specific solution for the ActiveInstalls tests and continue with the idea of starting XWiki from Maven.
    • To start ES using docker and in dev/testing mode: docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.1. However the minimal ES version provided is 5.2 and we need to verify if it's compatible with XWiki.
  • We need to start the DB, then generate the hibernate config, then generate the perm dir and provision the DB with the DataMojo, then start xwiki
    • However this is not easy... Some problems with maven phases to start twice the fabric8 plugin but can be worked around by using some contrived phases...
    • In addition this means that mvn docker:start doesn't work anymore...
    • Conclusion: we might need to use the XWiki REST API to install extensions after XWiki is started...
  • We need to find a way to have functional test modules depend on the module building the XWiki docker image somehow.
  • I have developed a maven plugin for installing an xwiki extension into a running XWiki. However this might not be the best strategy. Indeed if it fails then the docker containers won't be stopped. OTOH if we were to provision XWiki from the tests themselves, then the docker containers would be stopped if the provisioning failed (thanks to the usage of the failsafe maven plugin...).

Ideas

  • Maybe wrap the usage of DMP with our plugin so that we can reduce the verbosity of pom.xml files and reduce duplication
  • Try headless FF/Chome instead of xvfb
  • Use the new docker multistage feature to reduce image sizes

 


Tags:
Created by Vincent Massol on 2018/01/12 12:21
    

Get Connected