Description

Discussions

Discussion threads:

• Thread 1 (oldest): http://markmail.org/thread/c7ob3eth25hknhcy
• Thread 2 (newest): http://markmail.org/thread/gowutjcrfr62mq6t

Architecture

• An ElasticSearch (ES) instance installed on the xwiki.org domain somewhere.
• Local XWiki instances regularly (once per day?) send UUID + version information to that ElasticSearch instance using REST
• xwiki.org website displays counters by querying the ES instance using REST. Various views:
  • "Total # of XWiki instances": a counter that shows the total # of XWiki instances, whether they're active or not
  • "Total # of Active XWiki instances": a counter listing only active installs. Active is defined as having received a ping in the past 3 months
  • A table listing all extension ids and versions used and corresponding active install numbers. This allows to answer questions such as: how many XWiki instances are using the Bulleting Board extension and in which version
  • A table listing versions of XWiki used for active installs. This can be achieved by querying the version for one of the core extensions.
• An Admin page to let users opt out.
• Note that it's possible to run ES both in embedded more or as an external process.

Implementation Details

• Why use ES? Answer:
  • Separate from xwiki.org site and thus allows to handle the load better (separate machines for example). Also when xwiki.org XWiki is down, ES can still receive data
  • ES is scalable and powerful, has nice/powerful REST API, can handle large loads.
  • open source under an ASL license
• Allow configuring the ES URL in the xwiki configuration but for simplicity reasons, make it point by default to e.x.o and have an Apache redirect to redirect to activeinstalls.xwiki.org. This has 2 advantages:
  • Clients do not need to open 2 destinations in their firewalls (since they already need e.x.o it's already open)
  • On xwiki.org we do not need to expose a new IP/domain (activeinstalls.xwiki.org can remain internal as an IP only)
• Start a Ping Thread to send the ping data (infinite loop in run() with wait time of say, 1 day)
• At startup we create a table in the DB with the UUID so that it's stored and always stays the same even if the XWiki is upgraded
• Several maven modules:
  • xwiki-platform-activeinstalls/
    • xwiki-platform-activeinstalls-client/ : Client side module to send data, create table in DB for the UUID and UI to opt out
      • xwiki-platform-activeinstalls-client-api/ : Send data and create table in DB for the UUID
      • xwiki-platform-activeinstalls-client-ui/ : UI to opt out
    • xwiki-platform-activeinstalls-server/ : Server side module to compute/display stats from the ES store
      • xwiki-platform-activeinstalls-server-api/ : API to retrieve the data
      • xwiki-platform-activeinstalls-server-ui/ : Display results in wiki pages
• Data sent:
  • UUID
  • id + version of all extensions installed
• ES format for adding/updating an active install index:

  curl -XPUT "http://localhost:9200/installs/install/<UUID>" -d'
  {
    "formatVersion" : "<version e.g. 1.0>", <--- This allows changing the format in the future and still support older clients
    "date" : "<last ping date>",
    "distributionVersion" : "<version>",
    "extensions" : [
      {
        "id" : "<extension id>",
        "version" : "<extension version>"
      },
      ...
    ]
  }'

• We need to decide if we want to have the unique id feature be located in a separate module or part of the activeinstalls module.