archiva-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Lamy <ol...@apache.org>
Subject Re: maven-indexer / Lucene
Date Mon, 26 Jun 2017 01:49:54 GMT
Hi
graph/document storage could be convenient (but not possible with neo4j as
it's GPL license [1])
well we can add solr as an additional webapp with our jetty distribution
but this will be a pain for users who want to use tomcat or any other
servlet container...
we still need to investigate a new storage model :-)

Olivier
[1] https://neo4j.com/licensing/

On 25 June 2017 at 06:26, Martin <martin_s@apache.org> wrote:

> Yes, you are right. The lucene dependency causes a lot of trouble and will
> cause headaches with each version change of one of the dependencies.
> What are the requirements for a replacement?
> - We want to store hierarchical data?
> - We want to store metadata for nodes ?
> - Fulltext search (only metadata or for artifacts too?)
> - Blob / Artifact storage (I don't think so, but not so familiar with the
> archiva artifact model)?
>
> Maybe some graph database may be an alternative. Don't know if the license
> of
> neo4j is compatible to the apache license, and I think it brings lucene as
> dependency too. I will have a look.
> Problem is, if there is fulltext search needed, I think, for most of the
> frameworks we get a lucene dependency, if it's embedded.
>
> Other alternatives:
> - Implement fulltext search by our own (index of the metadata stored via
> the
> archiva api) and use the lucene dependency that comes from the
> maven-indexer
> - Jcr Oak with Solr. Solr is not embedded, must run as its own application
> (war).
>
> Greetings
>
> Martin
>
>
>
> Am Samstag, 24. Juni 2017, 14:05:26 CEST schrieb Olivier Lamy:
> > well this gonna be a pain.
> > IMHO we need to find a new alternative to jcr oak.
> > And something not using Lucene as it's a real pain to have different
> > librairies using lucene as they do not update in the same time (and
> Lucene
> > break backward compat so quickly...)
> > Any ideas? I'd like to have something embedded (but with a possible
> > external server configuration).
> > There is currently a Cassandra implementation. I was not satisfied about
> > performance but I guess I did that 4yo ago so can be improved for sure
> :-)
> > Maybe orientdb?
> > What else?
> >
> > On 24 June 2017 at 09:50, Olivier Lamy <olamy@apache.org> wrote:
> > > well the issue is non compatible version of Lucene for Maven Indexer
> and
> > > Oak (well I can try push a patch to Oak for upgrading...)
> > >
> > > On 24 June 2017 at 08:41, Olivier Lamy <olamy@apache.org> wrote:
> > >> Hi
> > >> Maven Indexer 6.0-SNAPSHOT doesn't need anymore plexus bridge.
> > >> I'm working on it in the branch ( feature/jcr_oak )
> > >> Not sure why but I have intermittent failure with store-jcr module.
> > >> I definitely agree on the upgrade.
> > >> Well we can simply detect it's not oak compatible and schedule a full
> > >> reindex (maybe with a message in logs and ui?)
> > >> But we need to be sure we can still read central index and not sure
> about
> > >> possible lucene conflict with oak and maven indexer.
> > >> We can work on this branch? (I created a Jenkins job for it
> > >> https://builds.apache.org/view/A-D/view/Archiva/job/archi
> > >> va-jcr-oak-branch/)
> > >> If you prefer master I would say no worries neither.
> > >> Something else to look at is upgrading maven-core etc...
> > >> Anyway
> > >> Cheers
> > >> Olivier
> > >>
> > >> On 22 June 2017 at 19:16, Martin <martin_s@apache.org> wrote:
> > >>> Hi,
> > >>>
> > >>> upgrading the maven indexer leads to some major changes.
> > >>> Lucene is used by maven-indexer and also by jackrabbit. Jackrabbit
> > >>> sticks to
> > >>> the old 3.x version and, as I see it, they will not move to a newer
> > >>> version.
> > >>> There is Jackrabbit Oak as alternative.
> > >>> I tried a proof of concept and could replace the jackrabbit
> > >>> implementation of
> > >>> metadata-store-jcr with a oak implementation. At least I got the unit
> > >>> tests of
> > >>> this module all to pass.
> > >>> But switching to Oak has some drawbacks:
> > >>> - The repository format changed and we must provide a way to migrate
> > >>> (either
> > >>> migrate the existing repository or create a new one by reindexing)
> > >>> - The lucene version used is newer but does not match to the version
> > >>> from the
> > >>> maven-indexer dependencies. There may come up some incompatibilities
> > >>> that are
> > >>> not solvable without using a modified version of one of the both. Or
> > >>> there may
> > >>> be the possibility to switch to solr (as separate component) and get
> rid
> > >>> of
> > >>> the lucene dependencies for jcr inside the archiva project.
> > >>>
> > >>> Switching to maven-indexer 6.0-SNAPSHOT means some changes too:
> > >>> - The Plexus-Sisu-Bridge does not work as before.
> > >>> - We must migrate from the NexusIndexer to the indexer API.
> > >>>
> > >>> So switching to the new indexer and oak means more work as expected
> and
> > >>> some
> > >>> risks regarding new incompatibility problems. And I think this
> cannot be
> > >>> done
> > >>> without broken master builds for some time period.
> > >>>
> > >>> So, what should we do? I think maven indexer is one of the core
> > >>> components of
> > >>> archiva, and we should utilize the 3.x-version to  migrate to the new
> > >>> indexer
> > >>> version, even if this means switching to jcr oak. Otherwise it would
> > >>> mean to
> > >>> stick to the old version for the next years.
> > >>> @Olivier, regarding the maven-indexer / sisu-Bridge API changes, I
> hope
> > >>> you
> > >>> can provide  useful help.
> > >>>
> > >>> I committed the PoC to the branch feature/jcr_oak. There are some
> > >>> modules
> > >>> where the tests do not pass (mainly because of the indexer API
> changes).
> > >>>
> > >>> Any comments?
> > >>>
> > >>> Cheers
> > >>>
> > >>> Martin
> > >>>
> > >>> Am Dienstag, 13. Juni 2017, 09:07:35 CEST schrieb Olivier Lamy:
> > >>> > forget it but we need to ensure we can read maven index files....
> > >>> >
> > >>> > On 13 June 2017 at 17:06, Olivier Lamy <olamy@apache.org>
wrote:
> > >>> > > Hi,
> > >>> > > Remember jackrabbit depends on Lucene as well so upgrading
Lucene
> > >>>
> > >>> can be a
> > >>>
> > >>> > > problem here.
> > >>> > > Regarding maven-indexer yes we can depend on a snapshot until
the
> > >>>
> > >>> release.
> > >>>
> > >>> > > I can release it ;-)
> > >>> > >
> > >>> > > On 13 June 2017 at 06:06, Martin <martin_s@apache.org>
wrote:
> > >>> > >> Hi,
> > >>> > >>
> > >>> > >> the lucene version depends on the maven indexer. But
I'm not
> sure
> > >>>
> > >>> about
> > >>>
> > >>> > >> the
> > >>> > >> current state of maven-indexer. The version has not changed
> since
> > >>>
> > >>> some
> > >>>
> > >>> > >> 2013.
> > >>> > >>
> > >>> > >> There are commits on the master branch since then, and
the
> lucene
> > >>>
> > >>> version
> > >>>
> > >>> > >> has
> > >>> > >> been changed too, but no releases were tagged.
> > >>> > >> Does it make sense to switch to the maven-indexer 6.0-SNAPSHOT?
> > >>> > >>
> > >>> > >> As I know there are new compact index formats with new
lucene
> > >>>
> > >>> versions
> > >>>
> > >>> > >> but I'm
> > >>> > >> not sure if this is relevant for the maven indexes.
> > >>> > >>
> > >>> > >> Cheers
> > >>> > >>
> > >>> > >> Martin
> > >>> > >
> > >>> > > --
> > >>> > > Olivier Lamy
> > >>> > > http://twitter.com/olamy | http://linkedin.com/in/olamy
> > >>
> > >> --
> > >> Olivier Lamy
> > >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> > >
> > > --
> > > Olivier Lamy
> > > http://twitter.com/olamy | http://linkedin.com/in/olamy
>
>
>


-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message