From version < 7.1 >
edited by Krzysztof Płachno
on 2017/06/09 15:11
To version < 8.1 >
edited by Krzysztof Płachno
on 2017/06/12 10:20
< >
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -120,15 +120,30 @@
120 120  ** GET /api/search/prop?[p1=v1,v2][&p2=v3][&repos=x[,y]]
121 121  ** to each uploaded package (to specific version or generally package as set of all versions) - you can specify in artifactory custom properties in form of Name:value and so you can later search packages according to those properties
122 122  
123 -
124 124  Bintray provides searching facilities only by name and/or description
125 125  
126 126  * GET /search/packages[?name=:name&desc=:desc]
127 127  * but those name and description are not imported from uploaded pom, but are inserted manually whilst uploading each package to Bintray
128 128  
129 -
130 130  === Questions: ===
131 131  
132 132  * Is it important to provide searching in ArtifactoryRepositoryManager and BintrayRepositoryManager? If so how to design it's logic using described above Artifactory/Bintray api?
133 133  * How to cope with the fact, that artifactory/bintray repo may contain also other artifacts apart from Xwiki Extension. (In maven connection is not the problem - since maven does not expose searching feature and to download extension you need to specify its full id)\\
134 134  ** Possible solution is that: on Artifactory/Bintray you may have plenty different, logically different maven repos. So prerequisite for connecting Artifactory/Bintray will be providing url to repo containing only xwiki valid extensions.
133 +
134 +(((
135 +
136 +
137 +== Update 12.06.2017 ==
138 +
139 +=== Summary of mail discussion 2 ===
140 +
141 +* at the moment we don't bother about Artifactory integration - it's less needed
142 +* I implement Bintray Connector as RepositoryManager and as Serchable (using this simply but sufficient searching options)
143 +* term Extension does not refer only to typical XWiki extensions that contain [email protected] or XWiki pages but generally all JARs, that can be later used e.g. in groovy script. Typical XWiki extension usually depend on some external JAR that need to be downloaded from Maven Central or JCenter
144 +* for the future I would be definitely useful to extend Extension Repository UI with an option to choose the repository from which we want to search extensions (e.g. search in Bintray JCenter, search in Extensions.org ect.)
145 +
146 +
147 +
148 +
149 +)))

 Feature
 Idea

Description

Artifactory and Bintray

Investigation report

There are two JFrog products: Artifactory and Bintray. At first glance they seem very similar. They both:

  • aggregate functionalities of different repositories: maven, npy, debian - and it's possible to connect them separately in native way (e.g. using maven - in pom or settings.xml we just provide URL to Artifactory/Bintray repo)
  • expose unified rest API for all those aggregated repositories and management operations - common for all repos

But the difference between Artifactory and Bintray (nicely described here: https://stackoverflow.com/a/38979659/4703098) is that 

  • Artifactory is a development-time tool - mainly focused on inside-company facilities like: sharing snapshots, proxying external repositories and caching artifacts, sharing 3rd party non-public artifacts, etc.
  • Bintray is a release, distribution-time tool - focused on hosting released version of artifacts for public or controlled usage, assuring load balancing, access...

Bintray and Artifactory can store in their repositories all types of binaries. But not all types have native support:

So we can upload to Bintray .egg package but it won't support connecting Bintray with pypi. (With Artifactory it's possible)

So from XWiki perspective of view

  • integration with Artifactory would make sense for those XWiki users that are quite big companies using XWiki for their own needs and producing many private XWiki extensions that they want to host in their Artifactory instance. The other beneficiary may be in the future XWiki community if it decides to migrate from Nexus to Artifactory.
  • integration with Bintray makes general sense - since there are public Bintray repositories like: https://jcenter.bintray.com, https://dl.bintray.com (moreover JCenter repository - according to what JFrog claims - is a superset of artifacts stored on maven central)

Problems:

  • I couldn't upload SNAPSHOT packages
  • It's strange - but when uploading package it's put in the root catalog instead of e.g. ''org/xwiki/contrib/....'' So if package should be later accessible from XWiki via bintray - you need to specify the "Target Repository Path" to proper value. Only then there will be flawless import from xwiki 
  • you need to remember - that after uploading packages form bintray UI - you need to publish them - only after publishing the package is available for download

Design questions

  1. Both: Artifactory and Bintray can be accessed natively using maven, npm, pypi (currently only artifactory) - is there a need/advantage of integrating those repositories via their REST API? (Assuming that either way we're going to develop specialized connectors with NPM and Pypi)
  • to check if connection is already working via maven connecor:
    • add following configuration in xwiki properties (it's a repo created by myself):
        ''extension.repositories=bintray:maven:http://dl.bintray.com/cristof/xwiki-test''
    • in your local xwiki instance you can import following extensions (that I uploaded to this bintray repo):
       ''org.xwiki.contrib:application-office365''  version: ''1.0.12345'' (current version of contrib application-office365 app with intentionally modified version)
       ''org.xwiki.contrib:hello-world-bintray-test''   version: ''0.1234'' (pure hello world component generated from xwiki archetype)
    • you can import them via ''Extension Manager'' or in ''Extension Repository App''

Update 09.06.2017

Summary of mail discussion

  • if rest apis of Bintary and Artifactory are simillar it'll be better to create one extension with functionality for connecting both
  • connecting Bintray will make sense mostly as connecting JCenter public repo and the best way will be to set such connection as default implementing ExtensionRepositorySource (like its done with extensions.xwiki.org or nexus.xwiki.org)
  • the advantage of using Rest API is that we can make use of searching features that are absent in native maven connection

Investigation report 2

General:

Potentially created ArtifactoryRepositoryManager BintrayRepositoryManager may implement apart from RespositoryManager interface, also Searchable interface. Currently only XWikiExtensionManager implements Searchable (AetherExtensionManager does not - Maven native operations does not provide searching options).

To implement RepositoryManager it's required to implement following operations:

  • resolve of extension (having its maven id and version e.g. org.xwiki.contrib:application-office365:1.0.123456
  • obtaining available versions of extension (having its maven id e.g. org.xwiki.contrib:application-office365

To implement Searchable it's required to implement following operation:

  • search of extensions giving pattern to search, offset and maximum search results

Artifactory and Bintray API

Implementation design with reference to Artifactory and Bintray API

Resolving extension

Obtaining available versions of extension

  • Artifactory
    • special path for obtaining versions
    • Path: GET /api/search/versions?[g=groupId][&a=artifactId][&v=version][&remote=0/1][&repos=x[,y]]
    • Example: GET /api/search/versions?g=org.xwiki.contrib&a=application-office365
  • Bintray
    • possible to obtain versions via searching packages (see docs for details)
    • Path: GET /search/packages/maven?g=:groupId&a=:artifactId&q=:query
    • Example: GET /search/packages/maven?g=org.xwiki.contrib&a=application-office365

Searching:

This is the biggest deal. Possible it should work as it does currently with XWikiExtensionManager.
How does searching in XWikiExtensionManager look like:XWikiExtensionManager utilizes Extension Repository Application API for searching: http://extensions.xwiki.org/xwiki/bin/view/Extension/Repository%20Module. Extension Repository Application doing search operation queries XWiki database (since during import of each extension to ERA - extension document page is created and extension's meta data is retrieved from pom and persisted in document xobjects in database). So the ERA search goes through extension metadata taken previously from pom properties like: or

Artifactory provides multiple package search methods (https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-SEARCHES) - for the scope of project following are interesting : 

  • by name (in Artifactory GUI - Quick search): GET /api/search/artifact?name=name[&repos=x[,y]]
    • the name parameter is supposed to somehow match the name of factual extension file e.g. '*office365*' matches 'application-office365-1.0.123456.xar'
    • but groupId is not taken into consideration e.g. name='*contrib*' will return no results
  • by maven coordinates (in Artifactory GUI - Maven GAVC Search)
    • GET /api/search/gavc?[g=groupId][&a=artifactId][&v=version][&c=classifier][&repos=x[,y]]
    • we don't need to specify all parameters (maven coordinates) e.g. we just put: g=*contrib* and it'll return all contrib extensions
  • by properties:
    • GET /api/search/prop?[p1=v1,v2][&p2=v3][&repos=x[,y]]
    • to each uploaded package (to specific version or generally package as set of all versions) - you can specify in artifactory custom properties in form of Name:value and so you can later search packages according to those properties

Bintray provides searching facilities only by name and/or description

  • GET /search/packages[?name=:name&desc=:desc]
  • but those name and description are not imported from uploaded pom, but are inserted manually whilst uploading each package to Bintray

Questions:

  • Is it important to provide searching in ArtifactoryRepositoryManager and BintrayRepositoryManager? If so how to design it's logic using described above Artifactory/Bintray api?
  • How to cope with the fact, that artifactory/bintray repo may contain also other artifacts apart from Xwiki Extension. (In maven connection is not the problem - since maven does not expose searching feature and to download extension you need to specify its full id)
    • Possible solution is that: on Artifactory/Bintray you may have plenty different, logically different maven repos. So prerequisite for connecting Artifactory/Bintray will be providing url to repo containing only xwiki valid extensions.

Update 12.06.2017

Summary of mail discussion 2

  • at the moment we don't bother about Artifactory integration - it's less needed
  • I implement Bintray Connector as RepositoryManager and as Serchable (using this simply but sufficient searching options)
  • term Extension does not refer only to typical XWiki extensions that contain [email protected] or XWiki pages but generally all JARs, that can be later used e.g. in groovy script. Typical XWiki extension usually depend on some external JAR that need to be downloaded from Maven Central or JCenter
  • for the future I would be definitely useful to extend Extension Repository UI with an option to choose the repository from which we want to search extensions (e.g. search in Bintray JCenter, search in Extensions.org ect.)


 


Get Connected