Changes for page [More extension repositories] Artifactory and Bintray
Last modified by Vincent Massol on 2024/02/26 17:57
Change comment:
There is no comment for this version
Summary
-
Page properties (1 modified, 0 added, 0 removed)
Details
- Page properties
-
- Content
-
... ... @@ -1,7 +1,7 @@ 1 1 == Artifactory and Bintray == 2 2 3 3 4 -=== Investigation r aport ===4 +=== Investigation report === 5 5 6 6 7 7 There are two JFrog products: **Artifactory **and **Bintray**. At first glance they seem very similar. They both: ... ... @@ -55,7 +55,6 @@ 55 55 === Investigation report 2 === 56 56 57 57 58 - 59 59 ==== General: ==== 60 60 61 61 Potentially created ArtifactoryRepositoryManager BintrayRepositoryManager may implement apart from RespositoryManager interface, also Searchable interface. Currently only XWikiExtensionManager implements Searchable (AetherExtensionManager does not - Maven native operations does not provide searching options). ... ... @@ -81,8 +81,9 @@ 81 81 82 82 ==== Implementation design with reference to Artifactory and Bintray API ==== 83 83 84 -Resolving extension 85 85 84 +**Resolving extension** 85 + 86 86 * dedicated paths for downloads 87 87 * Artifactory\\ 88 88 ** Path: GET /repo-key/path/to/artifact.ext ... ... @@ -91,7 +91,7 @@ 91 91 ** Path: GET https:~/~/dl.bintray.com/:subject/:repo/:file_path 92 92 ** [[https:~~/~~/dl.bintray.com/cristof/xwiki-test/org/xwiki/contrib/application-office365/1.0.12345/:application-office365-1.0.12345.xar>>url:https://dl.bintray.com/cristof/xwiki-test/org/xwiki/contrib/application-office365/1.0.12345/:application-office365-1.0.12345.xar]] 93 93 94 -Obtaining available versions of extension 94 +**Obtaining available versions of extension** 95 95 96 96 * Artifactory\\ 97 97 ** special path for obtaining versions ... ... @@ -102,13 +102,13 @@ 102 102 ** Path: GET /search/packages/maven?g=:groupId&a=:artifactId&q=:query 103 103 ** Example: GET /search/packages/maven?g=org.xwiki.contrib&a=application-office365 104 104 105 -Searching 105 +**Searching:** 106 106 107 107 This is the biggest deal. Possible it should work as it does currently with XWikiExtensionManager. 108 108 How does searching in XWikiExtensionManager look like:XWikiExtensionManager utilizes Extension Repository Application API for searching: [[http:~~/~~/extensions.xwiki.org/xwiki/bin/view/Extension/Repository%20Module>>url:http://extensions.xwiki.org/xwiki/bin/view/Extension/Repository%20Module]]. Extension Repository Application doing search operation queries XWiki database (since during import of each extension to ERA - extension document page is created and extension's meta data is retrieved from pom and persisted in document xobjects in database). So the ERA search goes through extension metadata taken previously from pom properties like: <name> or <description> 109 109 110 110 111 -Artifactory provides multiple package search methods ([[https:~~/~~/www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-SEARCHES>>url:https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-SEARCHES]]) - for the scope of project following are interesting : 111 +Artifactory provides multiple package search methods ([[https:~~/~~/www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-SEARCHES>>url:https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-SEARCHES||style="background-color: rgb(255, 255, 255);"]]) - for the scope of project following are interesting : 112 112 113 113 * by name (in Artifactory GUI - Quick search): GET /api/search/artifact?name=name[&repos=x[,y]]\\ 114 114 ** the name parameter is supposed to somehow match the name of factual extension file e.g. '*office365*' matches 'application-office365-1.0.123456.xar' ... ... @@ -125,8 +125,75 @@ 125 125 * GET /search/packages[?name=:name&desc=:desc] 126 126 * but those name and description are not imported from uploaded pom, but are inserted manually whilst uploading each package to Bintray 127 127 128 -Questions: 128 +=== Questions: === 129 129 130 130 * Is it important to provide searching in ArtifactoryRepositoryManager and BintrayRepositoryManager? If so how to design it's logic using described above Artifactory/Bintray api? 131 131 * How to cope with the fact, that artifactory/bintray repo may contain also other artifacts apart from Xwiki Extension. (In maven connection is not the problem - since maven does not expose searching feature and to download extension you need to specify its full id)\\ 132 132 ** Possible solution is that: on Artifactory/Bintray you may have plenty different, logically different maven repos. So prerequisite for connecting Artifactory/Bintray will be providing url to repo containing only xwiki valid extensions. 133 + 134 +((( 135 +== Update 12.06.2017 == 136 + 137 +=== Summary of mail discussion 2 === 138 + 139 +* at the moment we don't bother about Artifactory integration - it's less needed 140 +* I implement Bintray Connector as RepositoryManager and as Serchable (using this simply but sufficient searching options) 141 +* term Extension does not refer only to typical XWiki extensions that contain org.xwiki.component.annotation.@Components or XWiki pages but generally all JARs, that can be later used e.g. in groovy script. Typical XWiki extension usually depend on some external JAR that need to be downloaded from Maven Central or JCenter 142 +* for the future I would be definitely useful to extend Extension Repository UI with an option to choose the repository from which we want to search extensions (e.g. search in Bintray JCenter, search in Extensions.org ect.) 143 + 144 + 145 +== Update 13.06.2017 == 146 + 147 + 148 +=== Investigation report 3 === 149 + 150 +It turns out that bintray is only SAS software - you cannot download it and run on private server. The consequence is that - there's common api URL for all accounts on bintray, that is: 151 + 152 +* dl.bintray.com - for dowloading packages 153 +* api.bintray.com - for all other rest calls 154 + 155 +((( 156 +(Yes - the same api has different url for different things - cool - isn't it?) 157 +))) 158 + 159 +In documentation terminology - different bintray instances (accounts) are called :**subject** and different repositories on bintray instance (account) are called :**repo 160 +**Example: 161 + 162 +* the "jcenter" is a maven repository (:**repo**) running on Bintray instance called "bintray" (:subject) 163 +* according to rest api documentation if you want to download package via rest api you call following URL: 164 + GET [[https:~~/~~/dl.bintray.com/:subject/:repo/:file_path 165 + s>>url:https://dl.bintray.com/:subject/:repo/:file_pathso]]o in case of JCenter and the newest version guava library it'd be: 166 + GET [[https:~~/~~/dl.bintray.com/bintray/jcenter/com/google/guava/guava/22.0/guava-22.0.jar>>url:https://dl.bintray.com/bintray/jcenter/com/google/guava/guava/22.0/guava-22.0.jar]] 167 +* if you want to search a package via rest api: 168 +GET /search/packages[?name=:name&desc=:desc&subject=:subject&repo=:repo] 169 +(look! you do not need to specify nor :subject nor :repo and if they are absent searching goes through all public instances and repos on whole bintray SAS) 170 +in case we want to look for guava package in JCenter the call would be 171 +[[https:~~/~~/api.bintray.com/search/packages?name=guava&subject=bintray&repo=jcenter>>url:https://api.bintray.com/search/packages?name=guava&subject=bintray&repo=jcenter]] 172 + 173 +More over some REST API paths require authentication - like e.g. downloading (probably due to ddos protection). So to be able to download anything via REST you need to have an account on bintray.com (even a free one) and put Basic Auth Credentials in http requests. 174 +Look! We are still talking about accessing Bintray instance (account) that may have different repos (maven, npm, rpm, ...) via common rest api. Still we can access each of those repos directly in their native way. E.g. JCenter is a valid maven repo and we can connect it with maven client and without any authentication download packages. 175 +\\URL template for native connection with repos is dl.bintray.com/:subject/:repo 176 +E.g. 177 + 178 + 179 +* url for maven connection with my private bintray account is: [[http:~~/~~/dl.bintray.com/cristof/xwiki-test>>url:http://dl.bintray.com/cristof/xwiki-test||style="background-color: rgb(255, 255, 255);"]] 180 +* URL for JCenter maven connection due to it's popular has special one: [[http:~~/~~/jcenter.bintray.com>>url:http://jcenter.bintray.com/||style="background-color: rgb(255, 255, 255);"]] but [[http:~~/~~/dl.bintray.com/bintray/jcenter>>url:http://dl.bintray.com/bintray/jcenter||style="background-color: rgb(255, 255, 255);"]] works as well (tested) 181 + 182 +=== Summary and reasonings === 183 + 184 +* configuring bintray repository connection in xwiki.properties in the same format as other are enabled would not be intuitive e.g. 185 + extension.repositories=maven-xwiki:maven:http:~/~/nexus.xwiki.org/nexus/content/groups/public 186 + since Bintray is only SAS software - there's always the same url, and moreover there are different paths for different operations (see above) 187 +* the only thing to configure are the **:subject** and **:repo** which we want to connect and potentially the type of repo e.g. maven 188 +* this authentication in downloading via REST imho is a bit of nuisance - it's better to skip it using native connection to each type of repo (which does not require auth) and rest connection only for searching (which is not authorized) 189 + 190 +=== Proposal === 191 + 192 +* the proposal is to not introduce 'bintray' general connector but different connectors with particular type of repos that can be hosted on bintray e.g. 'bintray-maven', 'bintray-npm' and so. In this way we will utilize native connector for downloading and Bintray Rest API for searching 193 +* all those connectors will be packed together in one extension 194 +* to preserve current xwiki.properties configuration syntax, the config of bintray connection would look like following: 195 + jcenter:bintray-maven:http:~/~/dl.bintray.com/bintray/jcenter 196 + and the last two path segments will be parsed as :subject and :repo for usage in REST API calls 197 +* accoring to this in my bintray-maven connector I will implement from scratch only searching and other stuff will be done using maven (aether) connector from xwiki-commons 198 + 199 +)))