Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Mime-Version: 1.0 (Apple Message framework v752.3)
In-Reply-To: <103686.46633.qm@web30702.mail.mud.yahoo.com>
References: <103686.46633.qm@web30702.mail.mud.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <B41C33EF-9F0F-4526-95DE-C0590117D5B5@apache.org>
Content-Transfer-Encoding: 7bit
From: Grant Ingersoll <gsingers@apache.org>
Subject: Re: Multiple Indices vs Single Index
Date: Thu, 20 Sep 2007 11:55:01 -0400
To: java-user@lucene.apache.org

OK, I thought you meant your index would have in it the name of the  
second index and would thus do a two-stage retrieval.

At any rate, if you are saying your combined index with all the  
stored fields is ~3.4 GB I would think it would fit reasonably on the  
machine you have and perform reasonably.  Naturally, this depends on  
your application, your users, etc. and I can't make any guarantees,  
but I certainly recall others managing this size just fine.  See the  
many tips on improving searching and indexing on the Wiki (link at  
bottom in my signature) and do some profiling/testing.

When you said your tests were inconclusive, what tests have you  
done?  If you can, run the tests in a profiler to see where your  
bottlenecks are.

-Grant


On Sep 20, 2007, at 11:16 AM, Nikhil Chhaochharia wrote:

> I am sorry, it seems that I was not clear with what my problem is.   
> I will try to describe it again.
>
> My data is divided into 40 categories and at one time only one  
> category can be searched.  The GUI for the system will ask the user  
> to select the category from a drop-down.  Currently, I have a  
> separate index for every category.  The index sizes varies - one  
> category index is 10MB and another is 700MB.  Other index-sizes are  
> somewhere in between.
>
> I was wondering if it will be better to just have 1 large index  
> with all the 40 indices combined.  I do not need to do dual-queries  
> and my total index size (if I create a single index) is about  
> 3.4GB.  It will increase to maximum of 5-6 GB.  I am running this  
> on a dedicated machine with 8GB RAM.
>
> Unfortunately I do not have enough hardware to run both in parallel  
> and test properly.  Have just one server which is being used by  
> live users.  So it would be great if you could tell me whether I  
> should stick with my 40 indices or combine them into 1 index.  What  
> are the pros and cons of each approach ?
>
> Thanks,
> Nikhil
>
>
> ----- Original Message ----
> From: Grant Ingersoll <gsingers@apache.org>
> To: java-user@lucene.apache.org
> Sent: Thursday, 20 September, 2007 7:57:21 PM
> Subject: Re: Multiple Indices vs Single Index
>
> If I understand correctly, you want to do a two stage retrieval
> right?  That is, look up in the initial index (3.4 GB) and then do a
> second search on the sub index?  Presumably, you have to manage the
> Searchers, etc. for each of the sub-indexes as well as the big
> index.  This means you have to go through the hits from the first
> search, then route, etc. correct?
>
> Have you tried creating one single index with all the (stored)
> fields, etc?  Worst case scenario, assuming 1GB per index, is you
> would have a 40GB index, but my guess is index compression will
> reduce it more.  Since you are less than that anyway, have you tried
> just the straightforward solution?  Or do you have other requirements
> that force the sub-index solution?  Also, I am not sure it will work,
> but it seems worth a try.  Of course, this also depends on how much
> you expect your indexes to grow.
>
> Also, what was inconclusive about your tests?  Maybe you can describe
> more what you have tried to date?
>
> Cheers,
> Grant
>
> On Sep 20, 2007, at 3:50 AM, Nikhil Chhaochharia wrote:
>
>> Hi,
>>
>> I have about 40 indices which range in size from 10MB to 700MB.
>> There are quite a few stored fields.  To get an idea of the
>> document size, I have about 400k documents in the 700MB index.
>>
>> Depending on the query, I choose the index which needs to be
>> searched.  Each query hits only one index.  I was wondering if
>> creating a single index where every document will have the
>> indexname as a field will be more efficient.  I created such an
>> index and it was 3.4 GB in size.  My initial performance tests with
>> it are not conclusive.
>>
>> Also, what are the other points to be addressed while deciding
>> between 1 index and 40 indices.
>>
>> I have 8GB RAM on the machine.
>>
>>
>> Thanks,
>> Nikhil
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org