hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: KeyValue and BatchOperation
Date Fri, 29 May 2009 04:06:28 GMT
Answers interwoven below.

On Thu, May 28, 2009 at 7:01 PM, Sudipto Das <sudipto@cs.ucsb.edu> wrote:

> Hi,
> I am using HBase for development of a system for my research project. I
> have
> a few questions regarding some recent API and Class changes in HBase which
> I
> suppose are to be released in HBase 0.20.
> * I saw that for internal operations of HBase a new class
> org.apache.hadoop.hbase.KeyValue has replaced the usage of a lot of other
> classes (for example HLogEdit and HStoreKey), but till now, there is no
> change in the client API which still sends updates via BatchOperation and
> BatchUpdate which is then converted to KeyValue. So I am a bit confused
> whether to use KeyValue or the present BatchOperation class for
> communicating with the RegionServers. I am not using the HBase Client API,
> so I am not limited by that, so I was wondering which class would be a
> better choice to be compatible with 0.20.

You've caught TRUNK in a state of transition.   HBASE-1304 is the last
missing piece (HBASE-1234 added KeyValue and before that was change in our
store file format).  HBASE-1304 should be going in in the next week or so.
Its a fat patch that will deprecate current API in favor of a new, more
compact and comprehensive one.

The overarching motivation behind 0.20.0 refactoring is efficiency and
speed-up.  The server/client API will change in HBASE-1304 in a manner which
moves work out to the clients.  Server will probably be passing lists of
KeyValues to client to organize.

If you can wait a few more days, you'll have better idea of shape of things
to come (BatchUpdate and BatchOperation will be deprecated).

> * What is a good way of determining the HeapSize (in order to implement the
> hbase.io.HeapSize interface) for newly added classes? I saw that
> hbase.io.HeapSize has a few new constants provide sizes of some of the
> common types, but most HBase internal classes use an assigned value of
> HEAP_TAX, and I could not figure out how the value was obtained.

Value was obtained through study of heap changes creating particular class
instances in a controlled heap and by comparing with sizes reported by a
profiler and using an instrumented JVM that reports JVM deep sizeof (There
are a few. Here's one: http://www.javamex.com/classmexer/... though I  think
our Ryan was using something else).

Ask on list if you need help.  Some of the lads are getting good at this

> * For IPC, I could not pass a List of Writables since a List is not
> Writable. Is there any plan for adding a utility class (or is there already
> any such class available?) that can act as a List of Writables type and can
> be shipped across the network using IPC.

Our IPC is a customized version of the Hadoop IPC passing codes for
classnames to save on size of messages among other things.  HBase doesn't
need a generalized IPC as HBase does so we've cut it down.

If you want to add new types -- I don't think we have a list at moment, only
thing close are Map and array -- then you'll need to add a code.  If this is
not easy to do, then lets talk and make it so.  We want to make it so hbase
is subclassable or amendable and its proven so in some regard in that there
is the Transactional HBase done as a subclass but we for sure haven't made
universally so.  Help us out.

> * The HLog right now is very rigid in terms of what it accepts as Log
> Entries. Is there something inherent in Hadoop IO that prevents the Logger
> from accepting any Writable as log edit, rather than the mandatory KeyValue
> (or HLogEdit in 0.19.x). This will make the logger flexible and reusable
> for
> other uses. Apparently, Hadoop IO just needs writables. Is there any catch
> in a generic type for the Log Key and Log Edit?

As Value, I don't think so.  We've just made it so the value is just the
value -- not a container with some fluff and then the value (which was what
HLogEdit was).  If you want this change, just submit a patch and we'll
commit it.

> * As noted in the Bigtable paper, a single logger thread can become a
> bottleneck for an update intensive workload with Write Ahead Logging. I was
> wondering if an advanced logger will be available in some newer version of
> HBase? Advanced as in multi-threaded logger supporting asynchronous appends
> serialized using a common log sequence number.

This came up in recent discussions up on IRC.  We need to work on this from
both ends.  We've started to keep an accounting of how long writes take.
We're noticing that HDFS can stall quite frequently such that appends/syncs
can take seconds on occasion.    Because of this we started to talk up a
pool of log writers.  It wouldn't be hard to do.  We actually went ahead and
changed the HLogEdit key adding time-of-addition so that we can organize the
edits later by their arrival time.

On the other end, some work was done recently to multithread the log
splitting.  Its made a big improvement over what was there previous but is
insufficient serving a user-facing app out of hbase in real time.   The
multhreading can be compounded so both reading and writing are multithreaded
but more so, we need to make a system like that described in the bigtable
paper where the split of logs is distributed out across the cluster
MapReduce style.  This is a critical need but won't be done till 0.21.0 its

> Any comments and suggestions would be helpful.

Thanks for the throughtful questions.

Let us know how we can help get your project done.


> Thanks in advance.
> Regards
> Sudipto
> PhD Candidate
> Santa Barbara, CA 93106, USA
> http://www.cs.ucsb.edu/~sudipto <http://www.cs.ucsb.edu/%7Esudipto> <
> http://www.cs.ucsb.edu/%7Esudipto>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message