Return-Path: X-Original-To: apmail-archiva-dev-archive@www.apache.org Delivered-To: apmail-archiva-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4CEE310A93 for ; Wed, 26 Nov 2014 04:54:27 +0000 (UTC) Received: (qmail 897 invoked by uid 500); 26 Nov 2014 04:54:27 -0000 Delivered-To: apmail-archiva-dev-archive@archiva.apache.org Received: (qmail 844 invoked by uid 500); 26 Nov 2014 04:54:27 -0000 Mailing-List: contact dev-help@archiva.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@archiva.apache.org Delivered-To: mailing list dev@archiva.apache.org Received: (qmail 833 invoked by uid 99); 26 Nov 2014 04:54:27 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Nov 2014 04:54:27 +0000 Received: from mail-la0-f45.google.com (mail-la0-f45.google.com [209.85.215.45]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id C66311A048C for ; Wed, 26 Nov 2014 04:54:15 +0000 (UTC) Received: by mail-la0-f45.google.com with SMTP id gq15so1795696lab.4 for ; Tue, 25 Nov 2014 20:54:23 -0800 (PST) X-Received: by 10.152.26.131 with SMTP id l3mr32173907lag.49.1416977663572; Tue, 25 Nov 2014 20:54:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.158.16 with HTTP; Tue, 25 Nov 2014 20:54:03 -0800 (PST) In-Reply-To: References: From: Olivier Lamy Date: Wed, 26 Nov 2014 15:54:03 +1100 Message-ID: Subject: Re: Searching pom elements and generic metadata To: dev@archiva.apache.org Content-Type: text/plain; charset=UTF-8 release notes page is @ ./archiva-docs/src/site/apt/release-notes.apt.vm Not renaming but maybe copy values to the new column? Anyway I'm not sure if any users use cassandra :-) So maybe worth to mention in release note but not more? On 26 November 2014 at 09:49, Carlos Sanchez wrote: > Where would I put release notes? > For those using Cassandra they need to rename a column, not sure if it is > easy programmatically > > On Tue, Nov 25, 2014 at 10:40 PM, Olivier Lamy wrote: > >> Looks good. >> Ship it! :-) >> As you bump version to 2.2 there are some bulk updates to do in jira. >> >> -- >> Olivier >> On 25 Nov 2014 06:06, "Carlos Sanchez" wrote: >> >> > PR submitted as https://github.com/apache/archiva/pull/14 >> > >> > feedback welcome, there are some new methods there >> > >> > On Thu, Nov 20, 2014 at 11:09 AM, Carlos Sanchez >> > wrote: >> > >> > > I have added these REST methods with JCR implementations >> > > >> > > artifactsByProjectVersionMetadata/{key}/{value} >> > > artifactsByMetadata/{key}/{value} >> > > artifactsByProperty/{key}/{value} >> > > searchArtifacts/{text} >> > > searchArtifacts/{key}/{text} >> > > >> > > >> > > Docs here >> > > >> > >> https://github.com/carlossg/archiva/compare/carlossg:master...metadata-search >> > > >> > > Any suggestions on how to run these types of queries in Cassandra? >> > > >> > > >> > >> https://github.com/carlossg/archiva/compare/carlossg:master...metadata-search#diff-9de24dcf873b85547a60cd92cc2f6379R764 >> > > >> > > >> > > >> > > On Fri, Nov 7, 2014 at 11:19 PM, Carlos Sanchez >> > > wrote: >> > > >> > >> I have added search methods for any property, including facets >> > >> >> > >> restServices/archivaServices/browseService/searchArtifacts/key/text >> > >> restServices/archivaServices/browseService/searchArtifacts/text >> > >> >> > >> that search in JCR >> > >> >> > >> SELECT * FROM [archiva:projectVersion] AS projectVersion >> > >> LEFT OUTER JOIN [archiva:artifact] AS artifact ON >> > ISCHILDNODE(artifact, >> > >> projectVersion) >> > >> LEFT OUTER JOIN [archiva:facet] AS facet ON ISCHILDNODE(facet, >> > >> projectVersion) >> > >> WHERE (contains([projectVersion].*, $value) OR contains([facet].*, >> > >> $value)) >> > >> AND ISDESCENDANTNODE(artifact,'/repositories/internal/content/') >> > >> >> > >> But seems that performance is bad for a "normal" sized repo. Any >> > >> suggestions? >> > >> I'll try with subselects and tuning jcr, I would imagine that it >> should >> > >> work decently and there's something wrong with archiva. >> > >> >> > >> >> > >> >> > >> On Fri, Nov 7, 2014 at 11:17 AM, Olivier Lamy >> wrote: >> > >> >> > >>> On 7 November 2014 07:42, Carlos Sanchez wrote: >> > >>> > I got both getArtifactsByMetadata and getArtifactsByProperty >> working, >> > >>> > passing the repository >> > >>> > >> > >>> > >> > >>> >> > >> https://github.com/carlossg/archiva/compare/carlossg:abstract-metadata...metadata-search >> > >>> > >> > >>> > The problem I hit with not passing the repo is that >> > >>> > AbstractTestService.buildArtifacts needs it. Will look later. >> > >>> > >> > >>> > About adding the other pom fields to the search index, from what I >> > saw >> > >>> I >> > >>> > need to add them to MinimalArtifactInfoIndexCreator (or a new one) >> in >> > >>> the >> > >>> > maven-indexer project, and then archiva would just use that? >> > >>> >> > >>> In such case you will have to modify the lucene model in the maven >> > >>> indexer project ( can certainly breaks a lot :-) ). >> > >>> >> > >>> > >> > >>> > Are the fields indexed already searchable somewhere in archiva with >> > >>> field >> > >>> > selection? e.g. g:myGroupId or v:1.0 ? >> > >>> >> > >>> Not associated at this level but that's already in our repository. >> > >>> >> > >>> >> > >>> > >> > >>> > Thanks >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > On Wed, Nov 5, 2014 at 11:42 PM, Olivier Lamy >> > >>> wrote: >> > >>> > >> > >>> >> Hi, >> > >>> >> I would make repositoryId as a query param so will make it >> optional. >> > >>> >> If empty/null use the users repositories? >> > >>> >> >> > >>> >> On 6 November 2014 06:22, Carlos Sanchez >> wrote: >> > >>> >> > Thanks, got an initial implementation of getArtifactsByMetadata >> at >> > >>> >> > https://github.com/carlossg/archiva/compare/metadata-search >> > >>> >> > >> > >>> >> > I added the repositoryId param, like other methods, but I'm not >> > >>> sure if >> > >>> >> > that would be limiting the feature, searching in all repos would >> > be >> > >>> more >> > >>> >> > interesting. >> > >>> >> > >> > >>> >> > On Wed, Nov 5, 2014 at 12:28 PM, Olivier Lamy > > >> > >>> wrote: >> > >>> >> > >> > >>> >> >> basic Cassandra requests tru Hector client won't work. >> > >>> >> >> You will have to write some manual cql queries. >> > >>> >> >> >> > >>> >> >> On 5 November 2014 22:07, Carlos Sanchez >> > >>> wrote: >> > >>> >> >> > On Wed, Nov 5, 2014 at 11:54 AM, Olivier Lamy < >> > olamy@apache.org> >> > >>> >> wrote: >> > >>> >> >> > >> > >>> >> >> >> I would add that in the interface MetadataRepository, add a >> > >>> test in >> > >>> >> >> >> the tck and implements it for all providers :-) >> > >>> >> >> >> Maybe having 2 methods: >> > >>> >> >> >> * generic metadata: getArtifactsByMetadata( key, value) ( >> > maybe >> > >>> a Map >> > >>> >> >> >> for multiple value pair? will you test the full value or a >> > >>> matching?) >> > >>> >> >> >> >> > >>> >> >> > >> > >>> >> >> > is the matching syntax for cassandra an jcr different? >> > >>> >> >> > I mean if I pass ('foo', 'bar*') -or some sort of regex- to >> do >> > >>> >> matching >> > >>> >> >> is >> > >>> >> >> > it going to work only in jcr? >> > >>> >> >> > >> > >>> >> >> > >> > >>> >> >> > >> > >>> >> >> >> * pom contents: getArtifactsByProperty( key, value ) ( same >> > >>> >> questions ) >> > >>> >> >> >> >> > >>> >> >> >> To start I won't go with a Map and exact matching :-) (I >> > reckon >> > >>> pom >> > >>> >> >> >> content can be a pain with Cassandra impl) >> > >>> >> >> >> >> > >>> >> >> >> WDYT? >> > >>> >> >> >> >> > >>> >> >> >> Olivier >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> On 5 November 2014 21:45, Carlos Sanchez > > >> > >>> wrote: >> > >>> >> >> >> > Thanks for the pointers, so if I add a method to >> > >>> >> JcrMetadataRepository >> > >>> >> >> >> > >> > >>> >> >> >> > getArtifactsByProperty(key, value) >> > >>> >> >> >> > >> > >>> >> >> >> > similar to getArtifactsByDateRange, writing the correct >> JCR >> > >>> query, >> > >>> >> I >> > >>> >> >> >> could >> > >>> >> >> >> > access all that metadata? >> > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > In which of the rest services would it make sense to add >> > such >> > >>> a >> > >>> >> >> search? >> > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > On Wed, Nov 5, 2014 at 11:28 AM, Olivier Lamy < >> > >>> olamy@apache.org> >> > >>> >> >> wrote: >> > >>> >> >> >> > >> > >>> >> >> >> >> Hi, >> > >>> >> >> >> >> The current SearchService search in maven indexer >> available >> > >>> datas. >> > >>> >> >> >> >> All the data you want to search are not available here >> but >> > >>> in the >> > >>> >> >> >> >> MetadataRepository ( so jcr, file or the new Cassandra). >> > >>> >> >> >> >> And currently there is no search methods for that! >> > >>> >> >> >> >> So really good news to see you as a volunteer to >> implement >> > >>> this! >> > >>> >> :-) >> > >>> >> >> >> >> Let me know if you need any pointers. >> > >>> >> >> >> >> >> > >>> >> >> >> >> Cheers >> > >>> >> >> >> >> Olivier >> > >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> On 4 November 2014 06:33, Carlos Sanchez < >> > carlos@apache.org> >> > >>> >> wrote: >> > >>> >> >> >> >> > Hi, >> > >>> >> >> >> >> > >> > >>> >> >> >> >> > Two things I wasn't able to do and was going to >> implement >> > >>> >> (unless I >> > >>> >> >> >> >> missed >> > >>> >> >> >> >> > something): >> > >>> >> >> >> >> > >> > >>> >> >> >> >> > * search for generic metadata >> > >>> >> >> >> >> > * search for pom contents >> > >>> >> >> >> >> > >> > >>> >> >> >> >> > The way the search queries are built in SearchService >> > only >> > >>> looks >> > >>> >> >> for >> > >>> >> >> >> the >> > >>> >> >> >> >> > typical fields (artifactId, groupId,...), so that needs >> > to >> > >>> be >> > >>> >> >> changed, >> > >>> >> >> >> >> but >> > >>> >> >> >> >> > I don't know if the pom and generic metadata fields are >> > >>> even >> > >>> >> >> indexed, >> > >>> >> >> >> any >> > >>> >> >> >> >> > pointers? >> > >>> >> >> >> >> > >> > >>> >> >> >> >> > Looking to be able to search for "myMetadata:myValue" >> and >> > >>> >> something >> > >>> >> >> >> like >> > >>> >> >> >> >> > "developer:carlos*" through the rest api and ui. >> > >>> >> >> >> >> > >> > >>> >> >> >> >> > Thanks >> > >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> -- >> > >>> >> >> >> >> Olivier Lamy >> > >>> >> >> >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy >> > >>> >> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> -- >> > >>> >> >> >> Olivier Lamy >> > >>> >> >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy >> > >>> >> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> -- >> > >>> >> >> Olivier Lamy >> > >>> >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy >> > >>> >> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> -- >> > >>> >> Olivier Lamy >> > >>> >> http://twitter.com/olamy | http://linkedin.com/in/olamy >> > >>> >> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> Olivier Lamy >> > >>> http://twitter.com/olamy | http://linkedin.com/in/olamy >> > >>> >> > >> >> > >> >> > > >> > >> -- Olivier Lamy http://twitter.com/olamy | http://linkedin.com/in/olamy