lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Gershman <gregge...@yahoo.com>
Subject Re: Size + memory restrictions
Date Wed, 15 Feb 2006 00:41:32 GMT
You may consider incrementally adding documents to
your index; I'm not sure why there would be problems
adding to an existing index, but you can always add
additional documents.  You can optimize later to get
everything back into a single segment.

Querying is a different story; if you are using the
Sort API, you will need enough memory to store a full
sorting of your documents in memory.  If you're trying
to sort on a string or anything other than an int or
float, this could require a lot of memory.

I've used indices much bigger than 5 mil. docs/3.5 gb
with less than 4GB of RAM and had no problems.

Greg


--- Leon Chaddock <leonchaddock@macranet.co.uk> wrote:

> Hi,
> we are having tremendous problems building a large
> lucene index and querying 
> it.
> 
> The programmers are telling me that when the index
> file reaches 3.5 gb or 5 
> million docs the index file can no longer grow any
> larger.
> 
> To rectify this they have built index files in
> multiple directories. Now 
> apparently my 4gb memory is not enough to query.
> 
> Does this seem right to people or does anyone have
> any experience on largish 
> scale projects.
> 
> I am completely tearing my hair out here and dont
> know what to do.
> 
> Thanks
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message