hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject [common type encoding breakout] Re: HBase Hackathon @ Salesforce 05/06/2014 notes
Date Tue, 13 May 2014 00:01:02 GMT
Hey all,

There was a pow-wow at 5/6/14's HBase Hackathon after HBasecon about
coalescing to a common set of type encodings for simplified multi-system
interop.  Systems mentioned include apache phoenix, hive, kite, or spark
could move to. Present in the conversation were folks who have written
three different binary schemes: Ryan Blue /Adam Warrington (kite project
guys) Nick Dimiduk (behind o.a.h.h.types, though a little delayed), and
James Taylor (phoenix lead)

There is much similar work in each of these efforts and each scheme has
pros/cons and has lessons learned. The plan is to focus on the byte
encodings to build a spec that is a good middle ground pulling in the
effective pieces of each system as opposed to one particular existing

Before we start filing jira's, let me summarize so the discussion can
happen on list.

Ryan presented this spec based upon a survey and combination of the three


In the ensuing discussion, I believe we agree upon the main goals (having a
common encoding for typed data) but have a few points to hash out still.

Here's where where I believe there is a agreement:
* basic memcmp numeric encodings for ints, floats/doubles
** no unsigned vs signed int encoding -- just signed.
* fixed scale decimal type
* evolvability is highly desirable and thus tagged types of structs is
desirable.  seemed like agreement for protobuf (which meets criteria)
encoding for complex data types (records, arrays, lists with records, maps
in a single cell)
* no protobuf complex type encodings in rowkey (rowkey is like a struct but
memcmp is critical).
* varints in rowkeys likely not worth it because memcmp varints save little
* memcmp encodings for primitives in cells desired for phoenix (2ndary
* must support nulls in compound keys.
* this might be a separate module in hbase

Here's where the main discussion points that need follow up (or where I'm
not sure there was agreement):
* for compound key encoding with nulls, do we need to distinguish null from
""? (phoenix emulates oracle, where they are same)
* compound key encoding of string/byte[]'s (how to handle \0)
* do we include 1 byte and 2 byte ints?

Some things added since then.
* how to handle encodings of sql compound type like date (are they complex
or primitives?)
** updated suggestion for date encodings.
* Nick brings up some issues about the philosophy around
o.a.h.h.types.DataType.. As I understand it, this datatype api has
*extensibility* as the goal of being one api that could wrap many alternate
encodings of data for hbase.  The focus of the discussion initially was
around having one physical encoding usable by many systems for *interop*. I
think the two are orthogonal and compatible but we'd need some examples
(currently code inspection required) to see if they are actually
compatible.  (there might be some primitive type tagging required with the
current o.a.h.h.types implementation.)

Let's discuss the points that need followup and the new topics!

Jon and Ryan.

On Mon, May 12, 2014 at 10:32 AM, Stack <stack@duboce.net> wrote:

> Below are some rough notes taken during first 20 minutes of our
> hackathon/dev meetup last Tuesday (Your secretary had to abandon
> note-taking for white-boarding duty and failed to pass the baton).
> There were about ~50 folks in attendance with representatives
> from a variety of organizations.
> We started by promoting the items listed on the hackathon
> meetup page to the white board:
> http://files.meetup.com/1350316/IMG_20140506_190458.jpg
> The list of topics up for discussions were:
> + Agreeing on types among hbase-related projects (phoenix,
> kite, etc.).
> + Discussion around colocation of hbase master and meta
> and the master also doing regionserver services and whether
> we should retain the current topology (no regions on the
> master or backup master) for 1.0.
> + The ongoing 'consensus' work (putting zk usage behind
> a pluggable interface).
> + Transactions over HBase (Continuuity work, Xiaomi
> percolator, and VCNC's work).
> + Netty the server
> + HTrace
> + Multitenancy
> We started talking types.  It took a while to get going . Below are the raw
> notes:
> Static, dynamic or what?
> Lars Hofhansl: Is hbase a byte store? Should types be in there? If hbase
> knew
> about the type then could we exploit it internally.
> Should HBase know about types? Thought is that HBase could do some
> perf stuff if it was cognizant of types.  Maybe later do this or
> start out w/ a few common types first.
> At least one encoding strategy, work on this first.
> Make it so we don't design ourselves into a corner.
> Talking to Enis, Hive & Pig... what do these projects want?  Plug in codec?
> Kite project, tries to do a layer above byte store rather than in the fs.
> Encodings are not the same across storage engines for Kite.
> Does Parquet need to do float or integer?  But that is where it gets its
> perf advantage.
> Parquet does not necessarily work inside hbase.
> Kite as a lib?  That phoenix could use?
> Could add a phoenix encoding to Kite?  Yes.  But lets agree and then retire
> the avro encoding.
> Encoding is separate from schema.
> Ryan Blue: Don't want to say phoenix is on kite.  Just want to focus on
> encoding.
> James Taylor: Phoenix should be under kite, not on top of kite.
> Ryan: move to common encoding, and a lib to serialize....
> Jon Hsieh: where this could be used... phoenix encoding for kite... then
> all
> could use it writing out.
> Lars: What you doing at HP for type encoding.
> HP: Our code is in C.  All byte arrays.  We have a way of doing floats,
> etc.
> Ryan: But you want a serializing too?
> Lars: How we start in on this?
> Ryan: I sent out a doc. Ryan then presented this doc:
> https://docs.google.com/a/cloudera.com/document/d/15INOaxyifycpFxvB6xNdoj96JbOpslEywjkaoG8MURE/edit#heading=h.o1cgqtsqgqyg
> Pick just a few simple types and get agreement on these first?
> HP: We have seen in the past that this works for a while but then folks
> figure
> out that their complex types are slower than they should be and they start
> coming up w/ their own encodings as workarounds; now you are back to
> square one.... you need to version it all.
> Was proposed that we try and unify around a typing strategy that would work
> as
> the ONE way to do it in HBase with Phoenix going to try and come up on it
> first.
> To be continued up on the HBase dev list.
> Another proposal that had alot of nods in favor was that we not require
> 1.0 and 2.0 to be compatible.
> Notes ran out at this stage.
> St.Ack

// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message