lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leon Chaddock" <leonchadd...@macranet.co.uk>
Subject Re: Size + memory restrictions
Date Wed, 15 Feb 2006 09:28:00 GMT
Hi Greg,
Thanks. We are actually running against 4 segments of 4gb so about 20 
million docs. We cant merge the segments as their seems to be problems with 
out linux box , with having files over about 4gb. Not sure why that is.

If I was to upgrade to 8gb of ram does it seem likely this will double the 
amount of docs we can handle, or would this provide an exponential increase?

Thanks

Leon
----- Original Message ----- 
From: "Greg Gershman" <greggersh@yahoo.com>
To: <java-user@lucene.apache.org>
Sent: Wednesday, February 15, 2006 12:41 AM
Subject: Re: Size + memory restrictions


> You may consider incrementally adding documents to
> your index; I'm not sure why there would be problems
> adding to an existing index, but you can always add
> additional documents.  You can optimize later to get
> everything back into a single segment.
>
> Querying is a different story; if you are using the
> Sort API, you will need enough memory to store a full
> sorting of your documents in memory.  If you're trying
> to sort on a string or anything other than an int or
> float, this could require a lot of memory.
>
> I've used indices much bigger than 5 mil. docs/3.5 gb
> with less than 4GB of RAM and had no problems.
>
> Greg
>
>
> --- Leon Chaddock <leonchaddock@macranet.co.uk> wrote:
>
>> Hi,
>> we are having tremendous problems building a large
>> lucene index and querying
>> it.
>>
>> The programmers are telling me that when the index
>> file reaches 3.5 gb or 5
>> million docs the index file can no longer grow any
>> larger.
>>
>> To rectify this they have built index files in
>> multiple directories. Now
>> apparently my 4gb memory is not enough to query.
>>
>> Does this seem right to people or does anyone have
>> any experience on largish
>> scale projects.
>>
>> I am completely tearing my hair out here and dont
>> know what to do.
>>
>> Thanks
>>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> -- 
> Internal Virus Database is out-of-date.
> Checked by AVG Free Edition.
> Version: 7.1.375 / Virus Database: 267.15.0/248 - Release Date: 01/02/2006
>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message