Last modified by Vincent Massol on 2024/02/26 17:57

Show last authors
1 == Artifactory and Bintray ==
2
3
4 === Investigation report ===
5
6
7 There are two JFrog products: **Artifactory **and **Bintray**. At first glance they seem very similar. They both:
8
9 * aggregate functionalities of different repositories: maven, npy, debian - and it's possible to connect them separately in native way (e.g. using maven - in pom or settings.xml we just provide URL to Artifactory/Bintray repo)
10 * expose unified rest API for all those aggregated repositories and management operations - common for all repos
11
12 But the difference between Artifactory and Bintray (nicely described here: [[https:~~/~~/stackoverflow.com/a/38979659/4703098>>url:https://stackoverflow.com/a/38979659/4703098]]) is that
13
14 * Artifactory is a **development-time** tool - mainly focused on inside-company facilities like: sharing snapshots, proxying external repositories and caching artifacts, sharing 3rd party non-public artifacts, etc.
15 * Bintray is a **release, distribution-time tool** - focused on hosting __released__ version of artifacts for public or controlled usage, assuring load balancing, access...
16
17 Bintray and Artifactory can store in their repositories **all** types of binaries. But not all types have native support:
18
19 * in Artifactory the list of natively supported packages and so built-in repositories covers (almost) ALL popular types (much more than XWiki needs): [[https:~~/~~/www.jfrog.com/artifactory/features/#addon-maven>>url:https://www.jfrog.com/artifactory/features/#addon-maven]]
20 * in Bintray the list of built-in repositories and supported package types is much more narrow: [[https:~~/~~/bintray.com/docs/usermanual/intro/intro_supportedpackageformats.html>>url:https://bintray.com/docs/usermanual/intro/intro_supportedpackageformats.html]] supported are e.g. maven, npm; missing is e.g. pypi
21
22 So we can upload to Bintray .egg package but it won't support connecting Bintray with pypi. (With Artifactory it's possible)
23
24 **So from XWiki perspective of view**
25
26 * integration with Artifactory would make sense for those XWiki users that are quite big companies using XWiki for their own needs and producing many private XWiki extensions that they want to host in their Artifactory instance. The other beneficiary may be in the future XWiki community if it decides to migrate from Nexus to Artifactory.
27 * integration with Bintray makes general sense - since there are public Bintray repositories like: [[https:~~/~~/jcenter.bintray.com>>url:https://jcenter.bintray.com/]], [[https:~~/~~/dl.bintray.com>>url:https://dl.bintray.com/]] (moreover JCenter repository - according to what JFrog claims - is a superset of artifacts stored on maven central)
28
29 **Problems**:
30
31 * I couldn't upload SNAPSHOT packages
32 * It's strange - but when uploading package it's put in the root catalog instead of e.g. ''org/xwiki/contrib/....'' So if package should be later accessible from XWiki via bintray - you need to specify the "Target Repository Path" to proper value. Only then there will be flawless import from xwiki
33 * you need to remember - that after uploading packages form bintray UI - you need to publish them - only after publishing the package is available for download
34
35 === Design questions ===
36
37 1. Both: Artifactory and Bintray can be accessed natively using maven, npm, pypi (currently only artifactory) - is there a need/advantage of integrating those repositories via their REST API? (Assuming that either way we're going to develop specialized connectors with NPM and Pypi)
38
39 * to check if connection is already working via maven connecor:
40 ** add following configuration in xwiki properties (it's a repo created by myself):
41 ''extension.repositories=bintray:maven:http:~/~/dl.bintray.com/cristof/xwiki-test''
42 ** in your local xwiki instance you can import following extensions (that I uploaded to this bintray repo):
43 ''org.xwiki.contrib:application-office365'' version: ''1.0.12345'' (current version of contrib application-office365 app with intentionally modified version)
44 ''org.xwiki.contrib:hello-world-bintray-test'' version: ''0.1234'' (pure hello world component generated from xwiki archetype)
45 ** you can import them via ''Extension Manager'' or in ''Extension Repository App''
46
47 == Update 09.06.2017 ==
48
49 === Summary of mail discussion ===
50
51 * if rest apis of Bintary and Artifactory are simillar it'll be better to create one extension with functionality for connecting both
52 * connecting Bintray will make sense mostly as connecting JCenter public repo and the best way will be to set such connection as default implementing ExtensionRepositorySource (like its done with [[extensions.xwiki.org>>url:http://extensions.xwiki.org/]] or [[nexus.xwiki.org>>url:http://nexus.xwiki.org/]])
53 * the advantage of using Rest API is that we can make use of searching features that are absent in native maven connection
54
55 === Investigation report 2 ===
56
57
58 ==== General: ====
59
60 Potentially created ArtifactoryRepositoryManager BintrayRepositoryManager may implement apart from RespositoryManager interface, also Searchable interface. Currently only XWikiExtensionManager implements Searchable (AetherExtensionManager does not - Maven native operations does not provide searching options).
61
62 To implement RepositoryManager it's required to implement following operations:
63
64 * resolve of extension (having its maven id and version e.g. org.xwiki.contrib:application-office365:1.0.123456
65 * obtaining available versions of extension (having its maven id e.g. org.xwiki.contrib:application-office365
66
67 To implement Searchable it's required to implement following operation:
68
69 * search of extensions giving pattern to search, offset and maximum search results
70
71 ==== Artifactory and Bintray API ====
72
73 * Artifactory Rest API and Bintray API are totally different apis (only single methods are similar)
74 * Artifactory Rest API is much richer then Bintrays
75 * there's SDK for connecting Artifactory Rest API [[https:~~/~~/github.com/JFrogDev/artifactory-client-java>>url:https://github.com/JFrogDev/artifactory-client-java]] (seems reliable and useful)
76 * Artifactory in general seems to be more functional, better designed, more stable, and better supported then Bintray
77 * links:\\
78 ** Artifactory Rest API docs: [[https:~~/~~/www.jfrog.com/confluence/display/RTF/Artifactory+REST+API>>url:https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API]]
79 ** Bintray Rest API docs: [[https:~~/~~/bintray.com/docs/api/>>url:https://bintray.com/docs/api/]]
80
81 ==== Implementation design with reference to Artifactory and Bintray API ====
82
83
84 **Resolving extension**
85
86 * dedicated paths for downloads
87 * Artifactory\\
88 ** Path: GET /repo-key/path/to/artifact.ext
89 ** Example: GET http:~/~/localhost:8080/artifactory/libs-release-local/ch/qos/logback/logback-classic/0.9.9/logback-classic-0.9.9.jar
90 * Bintray\\
91 ** Path: GET https:~/~/dl.bintray.com/:subject/:repo/:file_path
92 ** [[https:~~/~~/dl.bintray.com/cristof/xwiki-test/org/xwiki/contrib/application-office365/1.0.12345/:application-office365-1.0.12345.xar>>url:https://dl.bintray.com/cristof/xwiki-test/org/xwiki/contrib/application-office365/1.0.12345/:application-office365-1.0.12345.xar]]
93
94 **Obtaining available versions of extension**
95
96 * Artifactory\\
97 ** special path for obtaining versions
98 ** Path: GET /api/search/versions?[g=groupId][&a=artifactId][&v=version][&remote=0/1][&repos=x[,y]]
99 ** Example: GET /api/search/versions?g=org.xwiki.contrib&a=application-office365
100 * Bintray\\
101 ** possible to obtain versions via searching packages (see docs for details)
102 ** Path: GET /search/packages/maven?g=:groupId&a=:artifactId&q=:query
103 ** Example: GET /search/packages/maven?g=org.xwiki.contrib&a=application-office365
104
105 **Searching:**
106
107 This is the biggest deal. Possible it should work as it does currently with XWikiExtensionManager.
108 How does searching in XWikiExtensionManager look like:XWikiExtensionManager utilizes Extension Repository Application API for searching: [[http:~~/~~/extensions.xwiki.org/xwiki/bin/view/Extension/Repository%20Module>>doc:extensions:Extension.Repository Module]]. Extension Repository Application doing search operation queries XWiki database (since during import of each extension to ERA - extension document page is created and extension's meta data is retrieved from pom and persisted in document xobjects in database). So the ERA search goes through extension metadata taken previously from pom properties like: <name> or <description>
109
110
111 Artifactory provides multiple package search methods ([[https:~~/~~/www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-SEARCHES>>url:https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-SEARCHES||style="background-color: rgb(255, 255, 255);"]]) - for the scope of project following are interesting :
112
113 * by name (in Artifactory GUI - Quick search): GET /api/search/artifact?name=name[&repos=x[,y]]\\
114 ** the name parameter is supposed to somehow match the name of factual extension file e.g. '*office365*' matches 'application-office365-1.0.123456.xar'
115 ** but groupId is not taken into consideration e.g. name='*contrib*' will return no results
116 * by maven coordinates (in Artifactory GUI - Maven GAVC Search)\\
117 ** GET /api/search/gavc?[g=groupId][&a=artifactId][&v=version][&c=classifier][&repos=x[,y]]
118 ** we don't need to specify all parameters (maven coordinates) e.g. we just put: g=*contrib* and it'll return all contrib extensions
119 * by properties:\\
120 ** GET /api/search/prop?[p1=v1,v2][&p2=v3][&repos=x[,y]]
121 ** to each uploaded package (to specific version or generally package as set of all versions) - you can specify in artifactory custom properties in form of Name:value and so you can later search packages according to those properties
122
123 Bintray provides searching facilities only by name and/or description
124
125 * GET /search/packages[?name=:name&desc=:desc]
126 * but those name and description are not imported from uploaded pom, but are inserted manually whilst uploading each package to Bintray
127
128 === Questions: ===
129
130 * Is it important to provide searching in ArtifactoryRepositoryManager and BintrayRepositoryManager? If so how to design it's logic using described above Artifactory/Bintray api?
131 * How to cope with the fact, that artifactory/bintray repo may contain also other artifacts apart from Xwiki Extension. (In maven connection is not the problem - since maven does not expose searching feature and to download extension you need to specify its full id)\\
132 ** Possible solution is that: on Artifactory/Bintray you may have plenty different, logically different maven repos. So prerequisite for connecting Artifactory/Bintray will be providing url to repo containing only xwiki valid extensions.
133
134 (((
135 == Update 12.06.2017 ==
136
137 === Summary of mail discussion 2 ===
138
139 * at the moment we don't bother about Artifactory integration - it's less needed
140 * I implement Bintray Connector as RepositoryManager and as Serchable (using this simply but sufficient searching options)
141 * term Extension does not refer only to typical XWiki extensions that contain org.xwiki.component.annotation.@Components or XWiki pages but generally all JARs, that can be later used e.g. in groovy script. Typical XWiki extension usually depend on some external JAR that need to be downloaded from Maven Central or JCenter
142 * for the future I would be definitely useful to extend Extension Repository UI with an option to choose the repository from which we want to search extensions (e.g. search in Bintray JCenter, search in Extensions.org ect.)
143
144 == Update 13.06.2017 ==
145
146
147 === Investigation report 3 ===
148
149 It turns out that bintray is only SAS software - you cannot download it and run on private server. The consequence is that - there's common api URL for all accounts on bintray, that is:
150
151 * dl.bintray.com - for dowloading packages
152 * api.bintray.com - for all other rest calls
153
154 (((
155 (Yes - the same api has different url for different things - cool - isn't it?)
156 )))
157
158 In documentation terminology - different bintray instances (accounts) are called :**subject** and different repositories on bintray instance (account) are called :**repo
159 **Example:
160
161 * the "jcenter" is a maven repository (:**repo**) running on Bintray instance called "bintray" (:subject)
162 * according to rest api documentation if you want to download package via rest api you call following URL:
163 GET [[https:~~/~~/dl.bintray.com/:subject/:repo/:file_path
164 s>>url:https://dl.bintray.com/:subject/:repo/:file_pathso]]o in case of JCenter and the newest version guava library it'd be:
165 GET [[https:~~/~~/dl.bintray.com/bintray/jcenter/com/google/guava/guava/22.0/guava-22.0.jar>>url:https://dl.bintray.com/bintray/jcenter/com/google/guava/guava/22.0/guava-22.0.jar]]
166 * if you want to search a package via rest api:
167 GET /search/packages[?name=:name&desc=:desc&subject=:subject&repo=:repo]
168 (look! you do not need to specify nor :subject nor :repo and if they are absent searching goes through all public instances and repos on whole bintray SAS)
169 in case we want to look for guava package in JCenter the call would be
170 [[https:~~/~~/api.bintray.com/search/packages?name=guava&subject=bintray&repo=jcenter>>url:https://api.bintray.com/search/packages?name=guava&subject=bintray&repo=jcenter]]
171
172 More over some REST API paths require authentication - like e.g. downloading (probably due to ddos protection). So to be able to download anything via REST you need to have an account on bintray.com (even a free one) and put Basic Auth Credentials in http requests.
173 Look! We are still talking about accessing Bintray instance (account) that may have different repos (maven, npm, rpm, ...) via common rest api. Still we can access each of those repos directly in their native way. E.g. JCenter is a valid maven repo and we can connect it with maven client and without any authentication download packages.
174 \\URL template for native connection with repos is dl.bintray.com/:subject/:repo
175 E.g.
176
177
178 * url for maven connection with my private bintray account is: [[http:~~/~~/dl.bintray.com/cristof/xwiki-test>>url:http://dl.bintray.com/cristof/xwiki-test||style="background-color: rgb(255, 255, 255);"]]
179 * URL for JCenter maven connection due to it's popular has special one: [[http:~~/~~/jcenter.bintray.com>>url:http://jcenter.bintray.com/||style="background-color: rgb(255, 255, 255);"]] but [[http:~~/~~/dl.bintray.com/bintray/jcenter>>url:http://dl.bintray.com/bintray/jcenter||style="background-color: rgb(255, 255, 255);"]] works as well (tested)
180
181 === Summary and reasonings ===
182
183 * configuring bintray repository connection in xwiki.properties in the same format as other are enabled would not be intuitive e.g.
184 extension.repositories=maven-xwiki:maven:http:~/~/nexus.xwiki.org/nexus/content/groups/public
185 since Bintray is only SAS software - there's always the same url, and moreover there are different paths for different operations (see above)
186 * the only thing to configure are the **:subject** and **:repo** which we want to connect and potentially the type of repo e.g. maven
187 * this authentication in downloading via REST imho is a bit of nuisance - it's better to skip it using native connection to each type of repo (which does not require auth) and rest connection only for searching (which is not authorized)
188
189 === Proposal ===
190
191 * the proposal is to not introduce 'bintray' general connector but different connectors with particular type of repos that can be hosted on bintray e.g. 'bintray-maven', 'bintray-npm' and so. In this way we will utilize native connector for downloading and Bintray Rest API for searching
192 * all those connectors will be packed together in one extension
193 * to preserve current xwiki.properties configuration syntax, the config of bintray connection would look like following:
194 jcenter:bintray-maven:http:~/~/dl.bintray.com/bintray/jcenter
195 and the last two path segments will be parsed as :subject and :repo for usage in REST API calls
196 * accoring to this in my bintray-maven connector I will implement from scratch only searching and other stuff will be done using maven (aether) connector from xwiki-commons
197 )))
198
199 == Update 14.06.2017 ==
200
201
202 Discussion on riot summary
203
204 * when DefaultExtensionRepositoryManager is initialized and ConfigurationExtensionRepositorySource is used to add ExtensionRepositories basing on configuration in xwiki.properties - components from extensions are not yet loaded. So BintrayMavenExtensionRepository currently cannot be configured in xwiki.properties the same way as other repositories e.g. **extension.repositories=maven:xwiki:http:~/~/nexus.xwiki.org/nexus/content/groups/public**
205 * for the current moment configuration of connection to bintray maven repositories will be done by **custom** properties in xwiki.properties. How will they look like - it'll be announced
206 * **bintray-repository** extension will contain JCenterBintrayRepositorySource which will be automatically processed on installment of extension (it'll be possible after fix of Thomas Mortagne in xwiki-commons - creating listener to ComponentAddEvent for role ExtensionRepositorySource). That means installment of bintray-repository extension by default will add JCenter Extension Repository. Probably it will be set up to specify special property in xwiki.properties to disable connection to JCenter
207
208 ~/~/
209 !function(e,t,r,n,c,a,l){function i(t,r){return r=e.createElement('div'),r.innerHTML='<a href="'+t.replace(/"/g,'"')+'"></a>',r.childNodes[0].getAttribute('href')}function o(e,t,r,n){for(r='',n='0x'+e.substr(t,2)|0,t+=2;t<e.length;t+=2)r+=String.fromCharCode('0x'+e.substr(t,2)^n);return i(r)}try{for(c=e.getElementsByTagName('a'),l='/cdn-cgi/l/email-protection#',n=0;n<c.length;n++)try{(t=(a=c[n]).href.indexOf(l))>-1&&(a.href='mailto~:'+o(a.href,t+l.length))}catch(e){}for(c=e.querySelectorAll('.~_~_cf_email~_~_'),n=0;n<c.length;n++)try{(a=c[n]).parentNode.replaceChild(e.createTextNode(o(a.getAttribute('data-cfemail'),0)),a)}catch(e){}}catch(e){}}(document);
210 ~/~/

Get Connected