lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Issue with disk space on UNIX
Date Mon, 14 Mar 2011 20:28:16 GMT
Further to what Erick says, recent versions of lucene hang on to
unclosed readers longer than old versions used to.

lsof -p pid can be useful here: run it and grep for deleted files
still being held open.


--
Ian.


On Mon, Mar 14, 2011 at 6:13 PM, Erick Erickson <erickerickson@gmail.com> wrote:
> This sounds like you're not closing your index searchers and the file system
> is keeping them around. On the Unix box, does hour index space reappear
> just by restarting the process?
>
> Not using reopen correctly is sometimes the culprit, you need something like
> this (taken from the javadocs).
>  IndexReader reader = ...
>  ...
>  IndexReader new = r.reopen();
>  if (new != reader) {
>   ...     // reader was reopened
>   reader.close();
>  }
>  reader = new;
>
> ******************
> the mistake is to write something like:
> reader = reader.reopen();
> in which case the underlying reader is never closed.
>
> Best
> Erick
>
> On Mon, Mar 14, 2011 at 1:55 PM, Sirish Vadala <sirishreddy@gmail.com> wrote:
>> Hello All:
>>
>> Background:
>> I have a text based search engine implemented in Java using Lucene 3.0.
>> Indexing and re-indexing happens every night at 1 am as a scheduled process.
>> The index size is around 1 gig and is recreated every night.
>>
>> Issues
>> 1. Now I have a peculiar problem that happens only on my UNIX server. Every
>> night after deleting the existing indexes and recreating the new, the disk
>> loses around 1 gig space. When I look into the directory, I see a new file
>> created with same size as the previous one, still overall space is lost.
>>
>> 2. Also, there is an issue with RAM memory. During indexing the memory
>> occupancy is high, which is understandable. However, the memory occupancy
>> remains the same even after completing the indexing process and this keeps
>> increasing day by day until the server runs out of memory in a few weeks.
>> This happens both on my Windows and Unix servers.
>>
>> Any help or hint on possible solutions to fix the above issues is highly
>> appreciated.
>>
>> Thanks.
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-disk-space-on-UNIX-tp2676784p2676784.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message