lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Lucene gobbling file descriptors
Date Tue, 01 Sep 2009 15:25:42 GMT
Late reply, but do what Michael said. I didn't understand that you hadso
many indexes, but opening/closing makes sense to me now, reusing
them wouldn't be very useful. Although I *can* imagine a "keep each open
for 5 minute" sort of rule being useful on the assumption that a user might
search several ways in a short interval, but I sure wouldn't go there unless
and until I had a performance issue to deal with....

In fact, I'd leave all the complex stuff (like, perhaps, queueing requests
after there were X outstanding ones being serviced) until I could
demonstrate
a need.

Although I would put in a bit of logging showing the max number of
concurrent requests just to try to get ahead of the "perfect storm"
scenario Michael mentioned.

Erick

On Tue, Sep 1, 2009 at 6:32 AM, Chris Bamford <Chris.Bamford@scalix.com>wrote:

> Hi Erick,
>
> >>Note that for search speed reasons, you really, really want to share your
> >>readers and NOT open/close for every request.
> I have often wondered about this - I hope you can help me understand it
> better in the context of our app, which is an email client:
>
> When one of our users receives email we index and store it so he (and only
> he) can search on it.  This means a separate index per user.  On large
> customer sites this can mean hundreds/thousands of indexes.  Sharing readers
> seems counter-intuitive, unless I am missing something.  What we do instead
> is that once a user performs a search, we keep his IndexReader open in case
> he searches again.  At present, we have no expiry on this mechanism, so they
> stay open indefinitely.  I'm a bit hazy on the underlying details but we
> have observed that the number of open fds jumps by around 10 each time a new
> user performs a search.  What would be a good strategy for managing this in
> your opinon?  Does it really make sense to keep the IndexReader open?  Would
> performance suffer that much if we did an open/close for each search?  Or
> would it perhaps be better to close open readers after a period of
> inactivity?
>
> Thanks for any wisdom / thoughts/ ideas.
>
> - Chris
>
>
>
> ----- Original Message -----
> From: Erick Erickson <erickerickson@gmail.com>
> Sent: Thu, 27/8/2009 4:49pm
> To: java-user@lucene.apache.org
> Subject: Re: Lucene gobbling file descriptors
>
> Note that for search speed reasons, you really, really want to share your
> readers and NOT open/close for every request.
> FWIW
> Erick
>
> On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <Chris.Bamford@scalix.com
> >wrote:
>
> > I'm glad its not normal.  That means we can fix it!  I will conduct a
> > review of IndexReader/Searcher open/close ops.
> >
> > Thanks!
> >
> > Chris
> >
> > ----- Original Message -----
> > From: Michael McCandless <lucene@mikemccandless.com>
> > Sent: Wed, 26/8/2009 2:26pm
> > To: java-user@lucene.apache.org
> > Subject: Re: Lucene gobbling file descriptors
> >
> > This is not normal.  As long as you are certain you close every
> > IndexReader/Searcher that you opened, the number of file descriptors
> > should stay "contained".
> >
> > Though: how many files are there in your index directory?
> >
> > Mike
> >
> > On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Chris.Bamford@scalix.com>
> > wrote:
> > > Hi there,
> > >
> > > I wonder if someone can help?  We have a successful Lucene app deployed
> > on Tomcat which works well.  As far as we can tell, our developers have
> > observed all the guidelines in the Lucene FAQ, but on some of our
> > installations, Tomcat eventually runs out of file descriptors and needs a
> > restart to clear it.  We know Lucene is the culprit because use lsof -p
> > <java PID> and the vast majority (usually tens of thousands) of files
> > reported are Lucene index files.
> > >
> > > I am hoping to get some tips on how this can be avoided.  Is it simply
> > the case that as time goes by, more and more descriptors are left open
> and
> > no matter how high ulimit is set, you will run out?  Or is there a policy
> of
> > recycling that we are failing to utilise properly?
> > >
> > > I am happy to provide more information, just don't know what at this
> > point!  Please ask....
> > >
> > > Thanks in advance
> > >
> > > - Chris
> > >
> > > Chris Bamford
> > > Senior Development Engineer
> > > Scalix
> > > chris.bamford@scalix.com
> > > Tel: +44 (0)1344 381814
> > > www.scalix.com
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message