hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: ZK rethink?
Date Sat, 11 Apr 2009 21:03:28 GMT

Hi Nitay,

We can, but actually I think it depends on how you want
to proceed: Up the timeout but negatively impact 
response times to failures? Or, move the ZK thread out
of Java into C/POSIX land to avoid problems with GC
related stalls? The latter option would make HBase
deployment more involved than just deploying jars here
and there so it does have that drawback, including the
need for some kind of fallback if the native libraries
containing the JNI glue are not installed or otherwise
fail to load. However as one of the guys on zookeeper-
dev rightly pointed out, because HBase mixes high data
throughput and object creation rates with low latency
requirements, for sake of stability the native threads
option is in my opinion the way to go (with appropriate
fallback). There may be other situations where this is
going to be desirable to avoid other problems during
long GC stalls. 

Additionally, another option is a tiny piece of native
code that, if present, and if specified in the
configuration, can do a mlockall() on startup. So now
in addition to latency sensitive threads running
outside of Java land, also the whole heap and
everything else is locked in RAM so the process will
be less affected if the node goes into swap. We are 
already recommending that users provision nodes to
hold all heap in RAM anyway, but we don't then also
establish the necessary contract with the OS to fully
take advantage of that. I note that -XX:+UseISM does
this (among other things) if the JVM is running on
Solaris. A mlockall() shim on Linux can accomplish
the same ends.

If we are going to want to manage region assignments
via ZK and do other fully distributed master-less
things going forward, getting latency sensitive
operations stable under high loads is going to be
really important. 

   - Andy

> From: Nitay <nitayj@gmail.com>
> Subject: Re: ZK rethink?
> To: hbase-dev@hadoop.apache.org, apurtell@apache.org
> Date: Thursday, April 9, 2009, 12:06 PM
> Andrew, shall we open up a JIRA to bump up the default
> timeout to 30 or 60
> seconds?


View raw message