hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chad Walters <Chad.Walt...@microsoft.com>
Subject Re: ZK rethink?
Date Sun, 12 Apr 2009 01:48:12 GMT
A couple thoughts:

-- Won't we soon be creating a lot less objects? This may reduce the pressure on the GC substantially.
Not that I am advocating we shouldn't do anything about this.

-- Have a Java ZK session handler with a longer timeout for ease-of-use and accessibility
to novices. Have a native C ZK session handler with a shorter timeout for folks who get serious
about things.

-- We definitely should avoid swapping - but doing an mlockall() seems like it might be going
too far. Is swapping actually part of the problems that Andrew has seen or are his issues
strictly related to the GC operating within physical memory bounds?

Chad

On 4/11/09 2:13 PM, "Ryan Rawson" <ryanobjc@gmail.com> wrote:

If you are using the jvm, it is not an option to swap ever. Full gcs will
destroy the performance.

We will need to limit the use of caching in hbase. Another option is a C
implementation of regionserver maybe.

On Apr 11, 2009 2:03 PM, "Andrew Purtell" <apurtell@apache.org> wrote:


Hi Nitay,

We can, but actually I think it depends on how you want
to proceed: Up the timeout but negatively impact
response times to failures? Or, move the ZK thread out
of Java into C/POSIX land to avoid problems with GC
related stalls? The latter option would make HBase
deployment more involved than just deploying jars here
and there so it does have that drawback, including the
need for some kind of fallback if the native libraries
containing the JNI glue are not installed or otherwise
fail to load. However as one of the guys on zookeeper-
dev rightly pointed out, because HBase mixes high data
throughput and object creation rates with low latency
requirements, for sake of stability the native threads
option is in my opinion the way to go (with appropriate
fallback). There may be other situations where this is
going to be desirable to avoid other problems during
long GC stalls.

Additionally, another option is a tiny piece of native
code that, if present, and if specified in the
configuration, can do a mlockall() on startup. So now
in addition to latency sensitive threads running
outside of Java land, also the whole heap and
everything else is locked in RAM so the process will
be less affected if the node goes into swap. We are
already recommending that users provision nodes to
hold all heap in RAM anyway, but we don't then also
establish the necessary contract with the OS to fully
take advantage of that. I note that -XX:+UseISM does
this (among other things) if the JVM is running on
Solaris. A mlockall() shim on Linux can accomplish
the same ends.

If we are going to want to manage region assignments
via ZK and do other fully distributed master-less
things going forward, getting latency sensitive
operations stable under high loads is going to be
really important.

  - Andy

> From: Nitay <nitayj@gmail.com>
> Subject: Re: ZK rethink?

> To: hbase-dev@hadoop.apache.org, apurtell@apache.org
> Date: Thursday, April 9, 2009, 12:06 PM

> Andrew, shall we open up a JIRA to bump up the default > timeout to 30 or
60 > seconds?


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message