hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geoff Hendrey" <ghend...@decarta.com>
Subject RE: scanner deadlock?
Date Mon, 12 Sep 2011 19:12:06 GMT
Hi - Will definitely switch to hotspot jre (as opposed to openjdk).

WRT scanner: I made sure scanner caching was set to 1, but I also played
with various levels of caching (up to 100). No impact on the problem.

WRT Put: write buffer size is 10*1024*1024 (10 MB). Each row is about
750KB, so I expect to see batching of order of 10 rows.

We are going to upgrade to 90.3 or 90.4. I am running my MR job with all
the setting changes recommended by Stack (except new JRE). Afterwards, I
will run again with the new JRE for comparison (want to change a few
things at a time, and test). After that we'll upgrade hbase and run our
tests again. And of course, I will keep the list posted on findings.

Guys: THANK YOU for all the help.

-geoff

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-Daniel Cryans
Sent: Monday, September 12, 2011 11:44 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?

> I thought that as long as I specified neither -client nor -server,
that
> Server Class detection would automatically invoke the "-server"
option.
>
>
http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
> s.html
>
> We are running 12-core AMD Opteron which is AMD64, so according to the
> guide above, -server is selected automatically. Please let me know if
> I've misunderstood this. We *definitely* want to be running hotspot!

It's two different JVMs, not a matter of using -client or -server
(which are just different configurations). What you are running is:

http://openjdk.java.net/

What most people run is:

http://www.oracle.com/us/technologies/java/index.html

>
> Regarding GC: we are generating GC logs for namenode, datanode, master
> and regionserver. We do see long GC from time to time. In fact, I
played
> with the mslab option, but didn't find significant improvement. We've
> seen times on the order of a minute in these logs, and have found no
way
> around it (spent countless days and nights experimenting with
different
> GC parameters, mslab, different heap sizes, etc).

Sometimes it's just a matter of how much data you have in flight.
That's why I mentioned scanner pre-caching (set via Scan.setCaching),
because it can potentially load a lot of rows into the RS's heap. More
concurrent scanners means also more data loaded into memory.

Are you also inserting at the same time? What's your write buffer size?

The discussion in this jira could be relevant:
https://issues.apache.org/jira/browse/HBASE-3813

A temporary fix got committed in 0.90.3 to make
ipc.server.max.queue.size configurable.

J-D

Mime
View raw message