lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ning Li <ning.li...@gmail.com>
Subject Re: Merging Solr Indexes
Date Wed, 01 Apr 2009 18:29:28 GMT
There is a jira issue on supporting index merge:
https://issues.apache.org/jira/browse/SOLR-1051.
But I agree with Otis that you should go with a single index first.

Cheers,
Ning


On Wed, Apr 1, 2009 at 12:06 PM, Otis Gospodnetic
<otis_gospodnetic@yahoo.com> wrote:
>
> Hi,
>
> Yes, you can write to the same index from multiple threads.  You still need to keep
track of the index size manually, whether you create 1 or N indices/cores.  I'd go with a
single index first.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: vivek sar <vivextra@gmail.com>
>> To: solr-user@lucene.apache.org
>> Sent: Wednesday, April 1, 2009 4:26:04 AM
>> Subject: Re: Merging Solr Indexes
>>
>> Thanks Otis. Could you write to same core (same index) from multiple
>> threads at the same time? I thought each writer would lock the index
>> so other can not write at the same time. I'll try it though.
>>
>> Another reason of putting indexes in separate core was to limit the
>> index size. Our index can grow up to 50G a day, so I was hoping
>> writing to smaller indexes would be faster in separate cores and if
>> needed I can merge them at later point (like end of day). I want to
>> keep daily cores. Isn't this a good idea? How else can I limit the
>> index size (beside multiple instances or separate boxes).
>>
>> Thanks,
>> -vivek
>>
>>
>> On Tue, Mar 31, 2009 at 8:28 PM, Otis Gospodnetic
>> wrote:
>> >
>> > Let me start with 4)
>> > Have you tried simply using multiple threads to send your docs to a single
>> Solr instance/core?  You should get about the same performance as what you are
>> trying with your approach below, but without the headache of managing multiple
>> cores and index merging (not yet possible to do programatically).
>> >
>> > Otis
>> > --
>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>> >
>> >
>> >
>> > ----- Original Message ----
>> >> From: vivek sar
>> >> To: solr-user@lucene.apache.org
>> >> Sent: Tuesday, March 31, 2009 1:59:01 PM
>> >> Subject: Merging Solr Indexes
>> >>
>> >> Hi,
>> >>
>> >>   As part of speeding up the index process I'm thinking of spawning
>> >> multiple threads which will write to different temporary SolrCores.
>> >> Once the index process is done I want to merge all the indexes in
>> >> temporary cores to a master core. For ex., if I want one SolrCore per
>> >> day then every index cycle I'll spawn 4 threads which will index into
>> >> some temporary index and once they are done I want to merge all these
>> >> into the day core. My questions,
>> >>
>> >> 1) I want to use the same schema and solrconfig.xml for all cores
>> >> without duplicating them - how do I do that?
>> >> 2) How do I merge the temporary Solr cores into one master core
>> >> programmatically? I've read the wiki on "MergingSolrIndexes", but I
>> >> want to do it programmatically (like in Lucene -
>> >> writer.addIndexes(..)) once the temporary indices are done.
>> >> 3) Can I remove the temporary indices once the merge process is done?
>> >> 4) Is this the right strategy to speed up indexing?
>> >>
>> >> Thanks,
>> >> -vivek
>> >
>> >
>
>

Mime
View raw message