lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Rochkind <rochk...@jhu.edu>
Subject RE: OutOfMemory GC: GC overhead limit exceeded - Why isn't WeakHashMap getting collected?
Date Tue, 14 Dec 2010 14:15:24 GMT
But the entirety of the old indexes (no longer on disk) wasn't cached in memory, right?  Or
is it?  Maybe this is me not understanding lucene enough. I thought that portions of the index
were cached in disk, but that sometimes the index reader still has to go to disk to get things
that aren't currently in caches.  If this is true (tell me if it's not!), we have an index
reader that was based on indexes that... are no longer on disk. But the index reader is still
open. What happens when it has to go to disk for info?

And the second replication will trigger a commit even if there are in fact no new files to
be transfered over to slave, because there have been no changes since the prior sync with
failed commit?
________________________________________
From: Upayavira [uv@odoko.co.uk]
Sent: Tuesday, December 14, 2010 2:23 AM
To: solr-user@lucene.apache.org
Subject: RE: OutOfMemory GC: GC overhead limit exceeded - Why isn't WeakHashMap getting collected?

The second commit will bring in all changes, from both syncs.

Think of the sync part as a glorified rsync of files on disk. So the
files will have been copied to disk, but the in memory index on the
slave will not have noticed that those files have changed. The commit is
intended to remedy that - it causes a new index reader to be created,
based upon the new on disk files, which will include updates from both
syncs.

Upayavira

On Mon, 13 Dec 2010 23:11 -0500, "Jonathan Rochkind" <rochkind@jhu.edu>
wrote:
> Sorry, I guess I don't understand the details of replication enough.
>
> So slave tries to replicate. It pulls down the new index files. It tries
> to do a commit but fails.  But "the next commit that does succeed will
> have all the updates." Since it's a slave, it doesn't get any commits of
> it's own. But then some amount of time later, it does another replication
> pull. There are at this time maybe no _new_ changes since the last failed
> replication pull. Does this trigger a commit that will get those previous
> changes actually added to the slave?
>
> In the meantime, between commits.. are those potentially large pulled new
> index files sitting around somewhere but not replacing the old slave
> index files, doubling disk space for those files?
>
> Thanks for any clarification.
>
> Jonathan
> ________________________________________
> From: yseeley@gmail.com [yseeley@gmail.com] On Behalf Of Yonik Seeley
> [yonik@lucidimagination.com]
> Sent: Monday, December 13, 2010 10:41 PM
> To: solr-user@lucene.apache.org
> Subject: Re: OutOfMemory GC: GC overhead limit exceeded - Why isn't
> WeakHashMap getting collected?
>
> On Mon, Dec 13, 2010 at 9:27 PM, Jonathan Rochkind <rochkind@jhu.edu>
> wrote:
> > Yonik, how will maxWarmingSearchers in this scenario effect replication?  If a slave
is pulling down new indexes so quickly that the warming searchers would ordinarily pile up,
but maxWarmingSearchers is set to 1.... what happens?
>
> Like any other commits, this will limit the number of searchers
> warming in the background to 1.  If a commit is called, and that tries
> to open a new searcher while another is already warming, it will fail.
>  The next commit that does succeed will have all the updates though.
>
> Today, this maxWarmingSearchers check is done after the writer has
> closed and before a new searcher is opened... so calling commit too
> often won't affect searching, but it will currently affect indexing
> speed (since the IndexWriter is constantly being closed/flushed).
>
> -Yonik
> http://www.lucidimagination.com
>

Mime
View raw message