lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vaijanath N. Rao" <vaijanath...@aol.com>
Subject Re: Does lucene support distributed indexing?
Date Tue, 29 Apr 2008 04:21:34 GMT
Hi all,

How about adding hadoop support for distributed indexing. If required I 
can start working on this. If Hadoop is the fesiable option.

Also what other technique one can think for doing distributed Indexing. 
Currently I am planning on extending the SolrJ to keep a map of where 
the document has gone and trying to get a distibuted Indexing.

--Thanks and Regards
Vaijanath


Otis Gospodnetic wrote:
> That's right - most of them are about distributed searching (hence my notes about sharding
being up to the app).  Hadoop's contrib/index is about dist indexing:
>
> "This contrib package provides a utility to build or update an index
> using Map/Reduce.
>
> A distributed "index" is partitioned into "shards". Each shard corresponds
> to a Lucene instance."
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
> ----- Original Message ----
>   
>> From: Chris Hostetter <hossman_lucene@fucit.org>
>> To: java-user@lucene.apache.org
>> Sent: Monday, April 28, 2008 7:53:43 PM
>> Subject: Re: Does lucene support distributed indexing?
>>
>>
>> : There are actually several distributed indexing or searching projects in 
>> : Lucene (the top-level ASF Lucene project, not Lucene Java), and it's 
>> : time to start thinking about the possibility of bringing them together, 
>> : finding commonalities, etc.
>>
>> I would actually argue that almost all of the examples you listed describe 
>> "distributed searching" to query multiple shards.
>>
>> As far as i know, none of them address the "distributed indexing" aspect: 
>> throw some raw data at the system and trust that it it will be indexed by 
>> one (or more) shard(s) in a way that "evenly" distributes the indexing 
>> "load"
>>
>>
>>
>> -Hoss
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message