lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vivek sar <vivex...@gmail.com>
Subject Re: Merging Solr Indexes
Date Wed, 01 Apr 2009 08:26:04 GMT
Thanks Otis. Could you write to same core (same index) from multiple
threads at the same time? I thought each writer would lock the index
so other can not write at the same time. I'll try it though.

Another reason of putting indexes in separate core was to limit the
index size. Our index can grow up to 50G a day, so I was hoping
writing to smaller indexes would be faster in separate cores and if
needed I can merge them at later point (like end of day). I want to
keep daily cores. Isn't this a good idea? How else can I limit the
index size (beside multiple instances or separate boxes).

Thanks,
-vivek


On Tue, Mar 31, 2009 at 8:28 PM, Otis Gospodnetic
<otis_gospodnetic@yahoo.com> wrote:
>
> Let me start with 4)
> Have you tried simply using multiple threads to send your docs to a single Solr instance/core?
 You should get about the same performance as what you are trying with your approach below,
but without the headache of managing multiple cores and index merging (not yet possible to
do programatically).
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: vivek sar <vivextra@gmail.com>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, March 31, 2009 1:59:01 PM
>> Subject: Merging Solr Indexes
>>
>> Hi,
>>
>>   As part of speeding up the index process I'm thinking of spawning
>> multiple threads which will write to different temporary SolrCores.
>> Once the index process is done I want to merge all the indexes in
>> temporary cores to a master core. For ex., if I want one SolrCore per
>> day then every index cycle I'll spawn 4 threads which will index into
>> some temporary index and once they are done I want to merge all these
>> into the day core. My questions,
>>
>> 1) I want to use the same schema and solrconfig.xml for all cores
>> without duplicating them - how do I do that?
>> 2) How do I merge the temporary Solr cores into one master core
>> programmatically? I've read the wiki on "MergingSolrIndexes", but I
>> want to do it programmatically (like in Lucene -
>> writer.addIndexes(..)) once the temporary indices are done.
>> 3) Can I remove the temporary indices once the merge process is done?
>> 4) Is this the right strategy to speed up indexing?
>>
>> Thanks,
>> -vivek
>
>

Mime
View raw message