lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: ConcurrentMergeScheduler may spawn MergeThreads forever
Date Tue, 28 Apr 2009 12:39:01 GMT
On Tue, Apr 28, 2009 at 8:28 AM, Shai Erera <serera@gmail.com> wrote:
> Every merge hit the exception, yes.
>
> And actually, the exceptions list is not used anywhere besides MT adding the
> exception to the list. That's why I was curious why it's there.

It's there so "anyUnhandledExceptions" can be called; we could add a
getter so that an app could query the CMS to see if there were
exceptions?  But, an app can also subclass CMS & override
handleMergeException to do something.

> I still think we should protect this case somehow, because even if it hits a
> disk-full exception, there's no point continuing to run infinitely?

The disk full could clear up, eg if something external was trying to
copy a massive file onto the same disk, IW could hit disk full, then
the copy would fail and remove the partially copied massive file, and
lots of space becomes available again.

> So maybe
> before spwaning the next thread, check the exceptions list and if it goes
> over a certain threshold (10?) fail?

But what does "fail" mean?  Stop doing any merges forever, for this IW
instance?  That seems dangerous.  EG maybe the massive merge will
fail, but little merges can proceed.  Also, stopping merging like this
spooks me because you might see a few exceptions, but then they stop
and you think all is good but in fact way too many segments are piling
up.

We already "fail" now, in that the thread that was doing the merge
will throw an exception up to the JRE's default thread exception
handler.  It's just that we then let MergePolicy select that merge
again.

I like the "sleep on exception" because 1) you will continue to see
that exceptions are being thrown, but 2) it won't saturate your CPU.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message