lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liviu Matei <liviu...@gmail.com>
Subject Re: Issue with Lucene 3.6.1 and MMapDirectory
Date Mon, 19 May 2014 09:15:55 GMT
Thank you very much to all of you the answers.
Uwe this is the strange thing that I am currently never closing the index
reader and opening a new one from 8 to 8 hours and I am noticing that crash
in indeed a highly concurrent environment.
The indexes reside in a NFS file system. And the location is shared between
multiple machines - on each machine running  multiple JVMs this is why I
mentioned shared mount.
Can you please tell me what parameters/settings should I check on the OS
side? I did ulimit and it returns unlimited. I will check for "
AlreadyClosedException" but I didn't saw that in the logs, but I will check
again.

2. With the next release I am trying to do the close of IndexReader. I
looked at SearchManager but what I am doing when doing a search I am also
doing a indexing of the search content in order to search in it also in
order to determine of score of the queries that I am constructing in the
itself content. With SearchManager if I am correct I cannot do that but
Clemens's approuch with ir.incRef(); should be ok also correct?

Thank you very much again,
Liviu




On Fri, May 16, 2014 at 11:55 AM, Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi,
>
> > Now if I don't
> > close the old index reader I am noticing increases of virtual memory with
> > every new reindex reopen (this should not be an issue on 64 bit Linux
> > correct - this is the configuration I am using and the indexes are on a
> > shared mount NTFS file system ).
>
> This always brings a virtual memory leak on all platforms (also Linux). In
> addition, files of older segments cannot be completely deleted anymore, so
> it also consumes disk space.
>
> > Can you please tell me if all this corruption is caused by the fact that
> I
> > am not closing the old IndexReader. But if I close if given that it is
> > share by multiple threads I will need to check each time before doing the
> > search if IndexReader is still open correct? Let's say in a thread I am
> > reopening the IndexReader and in another thread I am afterwards reusing
> the
> > old one in that case I should do the check correct? Or is there a smarter
> > mechanism in place.
>
> It is the other way round: If you not close the IndexReader it cannot
> crash (unless your JDK has a bug or somehow your filesystem [you mentioned
> shared... what does this mean?] forcefully unmaps the index files), it only
> happens if you close it! The issue here is: If you close the IndexReader
> and another thread is currently running a query, the above can happen,
> because the memory mapped buffer was forcefully unmapped by the
> MMapDirectory. Since Lucene 3.6.0, Lucene tries its best to prevent this
> crash from happening, but in high concurrency cases this may fail (because
> of missing synchronization, which would kill performance):
> http://issues.apache.org/jira/browse/LUCENE-3588
>
> In your case, in parallel to those crashes, you should also see
> "AlreadyClosedException", which is the root of the problem. It is just
> sometimes that the MMapDirectory code cannot correctly detect the already
> closed and crashes.
>
> So forcefully reopening indexreaders and closing the old ones, while
> queries are running is the wrong way to go. I would suggest to use
> SearcherManager, which can keep track of the indexreaders correctly.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
> > Sent: Wednesday, May 14, 2014 7:53 PM
> > To: java-user@lucene.apache.org
> > Subject: AW: Issue with Lucene 3.6.1 and MMapDirectory
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message