maven-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Cservenak <ta...@cservenak.net>
Subject Re: Using the Maven Indexer
Date Mon, 08 Dec 2014 20:26:21 GMT
Hi Eduard,

for additional information see:
http://jira.codehaus.org/browse/MINDEXER-81

Currently, the ArtifactInfo is hardwired, is not extensible.

Re available index for Central, it’s not the “minimal usable” 
the decision driver, but the SIZE of the index download instead.
We were experimenting with different creators, but the bandwidth
it took off (if you compared it to artifact downloads) was really huge.

As almost everyone uses MRMs, and they tend to “improve” the
basic GAV index Central publishes (ie. once Nexus caches a 
JAR file, it will “improve” the index with Classnames in the JAR too,
something Central does not publish).

artifactInfo#repoId should not return null, if asked via context.
If it does, there is a bug lurking somewhere.

Currently the “extra info” path is viable, but that would create a lot
of cruft around indexer classes…..





-- 
Thanks,
~t~

On 8 Dec 2014 at 17:15:08, Eduard Moraru (enygma2002@gmail.com) wrote:

Hi,  

I have a new challenge for your maven-indexer expertise :)  

What about adding additional information to the local index? I see the  
default indexers (min, etc..) produce really minimal information. The  
problem is that everybody is using these default indexers and all the  
available indexes (maven central, etc) offer very little information that  
you can actually use to make the index useful in an application outside of  
really basic name, description, group, artifact, etc queries.  

For instance, if I would want to add author information (to query by  
author) or dependency information (to perform compatibility checks against  
an installation/group of installed artifacts) or anything else for the  
matter, what would be the recommended approach?  

From what I have currently researched, I see 2 options:  

1) Have a custom IndexCreator that uses the updateDocument(ArtifactInfo  
artifactInfo, Document document) method to fetch (HTTP GET) get the pom.xml  
by using information from the artifactInfo object (repository, groupID,  
artifactId, classifier, version, etc.) so that the resulting document  
contains the extra information. It seems that IndexCreators are used a lot  
more than they are advertised in the descriptions, not only for indexing  
new items, but also when converting between ArtifactInfo objects and Lucene  
Documents.  

1.1) I had initially started going on this pat, but then I realized that  
the artifactInfo that I receive in this method does not provide basic  
information (i.e. artifactInfo.getRepository() always returns null ;-( ) It  
would be awesome if information like context and/or repository would be  
added to the artifactInfo object (maybe in  
IndexUtils.constructArtifactInfo( Document doc, IndexingContext context )  
?), the same way the ArtifactInfo.UINFO and ArtifactInfo.LAST_MODIFIED  
fields are handled specially and explicitly added to a new Document that is  
passed to the IndexCreators.  

2) Handle this separately from maven indexer's work, and do it right after  
index/update operations, i.e. let maven-indexer update the local index with  
information from the remote index and then start manipulating the  
underlying Lucene index by adding information retrieved from the network  
(HTTP GET) from the remote repositoy's POM files. In a rough pseudocode,  
something like:  

indexer.update(repoX);  
indexer.getAllIndexedArtifacts().forEach(artifact ->  
var extraData = getExtraData(repoX, artifact);  
var indexer.getLuceneIndex().add(artifact, extraData)  
);  

3) Any other suggestions?  

My ultimate goal is (besides basic name/description queries) to be able to  
perform compatibility queries on artifacts coming from multiple  
repositories, so I need to find a solution to add this missing infrmation  
(artifact dependencies, and maybe more).  

As previously, your help and suggestions are most welcomed.  

Thanks,  
Eduard  

On Wed, Nov 26, 2014 at 1:22 PM, Eduard Moraru <enygma2002@gmail.com> wrote:  

>  
>  
> On Tue, Nov 25, 2014 at 12:22 PM, Tamas Cservenak <tamas@cservenak.net>  
> wrote:  
>  
>> Hi there,  
>>  
>> 1) yes, indexing context retains the artefact “origin” (ie. repo), so you  
>> need context per origin. Sadly, the 1 index per context is current  
>> limitation of maven indexer, but this problem is known. Created  
>> http://jira.codehaus.org/browse/MINDEXER-93  
>>  
>> 2) Yes, merged context is basically delegating to member contexts. under  
>> the hud, it uses Lucene’s MultiReader to actually perform the search.  
>>  
>  
> I have solved the search problem for now by using the SearchEngine  
> component and issuing an IteratorSearchRequest on a list of  
> IndexingContexts to get paginated results. Will have to see how that works  
> on the long run.  
>  
> Thanks,  
> Eduard  
>  
>  
>> Re ranging, there are already issues (or problem spread across multiple  
>> issues), most notably this one  
>> http://jira.codehaus.org/browse/MINDEXER-8  
>>  
>> 3) I think yes. Currently, indexer is being transitioned from Plexus to  
>> JSR330, and as you see in examples, it should work with any container  
>> supporting it. re “manually wiring”, in latest releases you might be able  
>> to do it, but in older ones probably not, as Plexus supported field  
>> injection only, and some of those member was not exposed via getter/setter.  
>> See  
>> http://jira.codehaus.org/browse/MINDEXER-80  
>>  
>>  
>> --  
>> Thanks,  
>> ~t~  
>>  
>> On 21 Nov 2014 at 18:08:26, Eduard Moraru (enygma2002@gmail.com) wrote:  
>>  
>> Hi,  
>>  
>> I have recently started playing with the maven indexer [1], following the  
>> examples [2], and I have some questions (since AFAIS, documentation is  
>> practically unexistent on the matter):  
>>  
>> 1) From what I can understand, you need an IndexingContext for each  
>> repository you plan to index. This makes you end up with n lucene indexes,  
>> one for each repository. Is there any way that I could have just 1 lucene  
>> index, with all my repositories indexed in the same place? If the main  
>> purpose is searchig, why scatter the indexed information across n indexes  
>> and make the whole process dificult? Maybe I`m missing something.  
>>  
>> 2) On the same line as the first question, when it comes to searching, it  
>> seems that I can use a MergedIndexingContext to perform a search on  
>> multiple (all) indexed repositories (IndexingContexts). How does this  
>> merge  
>> the search results? I assume it takes each lucene index and queries it  
>> individually, but this probably means that the lucene scores of these  
>> merged results are completely messed up and ureliable, right?  
>> Any suggestions on how to properly perform search over multiple indexed  
>> repositories?  
>>  
>> 3) About the Plexus Container: Am I forced to initialize and use one, or  
>> can I/should manually instantiate the default implementations and use them  
>> instead?  
>>  
>> I`ll probably come up with more questions along the way, hope someone will  
>> find the time to guide me on the right path.  
>>  
>> Thanks,  
>> Eduard  
>>  
>> ----------  
>> [1] https://github.com/apache/maven-indexer/  
>> [2]  
>>  
>> https://github.com/apache/maven-indexer/tree/master/indexer-examples/indexer-examples-basic
 
>>  
>  
>  

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message