hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject HBase Hackathon @ Salesforce 05/06/2014 notes
Date Mon, 12 May 2014 17:32:54 GMT
Below are some rough notes taken during first 20 minutes of our
hackathon/dev meetup last Tuesday (Your secretary had to abandon
note-taking for white-boarding duty and failed to pass the baton).

There were about ~50 folks in attendance with representatives
from a variety of organizations.

We started by promoting the items listed on the hackathon
meetup page to the white board:


The list of topics up for discussions were:

+ Agreeing on types among hbase-related projects (phoenix,
kite, etc.).
+ Discussion around colocation of hbase master and meta
and the master also doing regionserver services and whether
we should retain the current topology (no regions on the
master or backup master) for 1.0.
+ The ongoing 'consensus' work (putting zk usage behind
a pluggable interface).
+ Transactions over HBase (Continuuity work, Xiaomi
percolator, and VCNC's work).
+ Netty the server
+ HTrace
+ Multitenancy

We started talking types.  It took a while to get going . Below are the raw

Static, dynamic or what?

Lars Hofhansl: Is hbase a byte store? Should types be in there? If hbase
about the type then could we exploit it internally.

Should HBase know about types? Thought is that HBase could do some
perf stuff if it was cognizant of types.  Maybe later do this or
start out w/ a few common types first.

At least one encoding strategy, work on this first.

Make it so we don't design ourselves into a corner.

Talking to Enis, Hive & Pig... what do these projects want?  Plug in codec?

Kite project, tries to do a layer above byte store rather than in the fs.

Encodings are not the same across storage engines for Kite.

Does Parquet need to do float or integer?  But that is where it gets its
perf advantage.

Parquet does not necessarily work inside hbase.

Kite as a lib?  That phoenix could use?

Could add a phoenix encoding to Kite?  Yes.  But lets agree and then retire
the avro encoding.

Encoding is separate from schema.

Ryan Blue: Don't want to say phoenix is on kite.  Just want to focus on

James Taylor: Phoenix should be under kite, not on top of kite.

Ryan: move to common encoding, and a lib to serialize....

Jon Hsieh: where this could be used... phoenix encoding for kite... then all
could use it writing out.

Lars: What you doing at HP for type encoding.

HP: Our code is in C.  All byte arrays.  We have a way of doing floats, etc.

Ryan: But you want a serializing too?

Lars: How we start in on this?

Ryan: I sent out a doc. Ryan then presented this doc:


Pick just a few simple types and get agreement on these first?

HP: We have seen in the past that this works for a while but then folks
out that their complex types are slower than they should be and they start
coming up w/ their own encodings as workarounds; now you are back to
square one.... you need to version it all.

Was proposed that we try and unify around a typing strategy that would work
the ONE way to do it in HBase with Phoenix going to try and come up on it
To be continued up on the HBase dev list.

Another proposal that had alot of nods in favor was that we not require
1.0 and 2.0 to be compatible.

Notes ran out at this stage.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message