hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vrodio...@carrieriq.com>
Subject RE: HBase Hackathon @ Salesforce 05/06/2014 notes
Date Mon, 12 May 2014 23:41:07 GMT
Apache mail server is eventually consistent system (most of the time), that is why sometimes
we receive replies before the original mail and sometimes do not receive them at all. Reminds
me Cassandra.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
Sent: Monday, May 12, 2014 10:32 AM
To: HBase Dev List
Subject: HBase Hackathon @ Salesforce 05/06/2014 notes

Below are some rough notes taken during first 20 minutes of our
hackathon/dev meetup last Tuesday (Your secretary had to abandon
note-taking for white-boarding duty and failed to pass the baton).

There were about ~50 folks in attendance with representatives
from a variety of organizations.

We started by promoting the items listed on the hackathon
meetup page to the white board:

http://files.meetup.com/1350316/IMG_20140506_190458.jpg

The list of topics up for discussions were:

+ Agreeing on types among hbase-related projects (phoenix,
kite, etc.).
+ Discussion around colocation of hbase master and meta
and the master also doing regionserver services and whether
we should retain the current topology (no regions on the
master or backup master) for 1.0.
+ The ongoing 'consensus' work (putting zk usage behind
a pluggable interface).
+ Transactions over HBase (Continuuity work, Xiaomi
percolator, and VCNC's work).
+ Netty the server
+ HTrace
+ Multitenancy

We started talking types.  It took a while to get going . Below are the raw
notes:

Static, dynamic or what?

Lars Hofhansl: Is hbase a byte store? Should types be in there? If hbase
knew
about the type then could we exploit it internally.

Should HBase know about types? Thought is that HBase could do some
perf stuff if it was cognizant of types.  Maybe later do this or
start out w/ a few common types first.

At least one encoding strategy, work on this first.

Make it so we don't design ourselves into a corner.

Talking to Enis, Hive & Pig... what do these projects want?  Plug in codec?

Kite project, tries to do a layer above byte store rather than in the fs.

Encodings are not the same across storage engines for Kite.

Does Parquet need to do float or integer?  But that is where it gets its
perf advantage.

Parquet does not necessarily work inside hbase.

Kite as a lib?  That phoenix could use?

Could add a phoenix encoding to Kite?  Yes.  But lets agree and then retire
the avro encoding.

Encoding is separate from schema.

Ryan Blue: Don't want to say phoenix is on kite.  Just want to focus on
encoding.

James Taylor: Phoenix should be under kite, not on top of kite.

Ryan: move to common encoding, and a lib to serialize....

Jon Hsieh: where this could be used... phoenix encoding for kite... then all
could use it writing out.

Lars: What you doing at HP for type encoding.

HP: Our code is in C.  All byte arrays.  We have a way of doing floats, etc.

Ryan: But you want a serializing too?

Lars: How we start in on this?

Ryan: I sent out a doc. Ryan then presented this doc:

https://docs.google.com/a/cloudera.com/document/d/15INOaxyifycpFxvB6xNdoj96JbOpslEywjkaoG8MURE/edit#heading=h.o1cgqtsqgqyg

Pick just a few simple types and get agreement on these first?

HP: We have seen in the past that this works for a while but then folks
figure
out that their complex types are slower than they should be and they start
coming up w/ their own encodings as workarounds; now you are back to
square one.... you need to version it all.

Was proposed that we try and unify around a typing strategy that would work
as
the ONE way to do it in HBase with Phoenix going to try and come up on it
first.
To be continued up on the HBase dev list.

Another proposal that had alot of nods in favor was that we not require
1.0 and 2.0 to be compatible.

Notes ran out at this stage.

St.Ack

Confidentiality Notice:  The information contained in this message, including any attachments
hereto, may be confidential and is intended to be read only by the individual or entity to
whom this message is addressed. If the reader of this message is not the intended recipient
or an agent or designee of the intended recipient, please note that any review, use, disclosure
or distribution of this message or its attachments, in any form, is strictly prohibited. 
If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com
and delete or destroy any copy of this message and its attachments.

Mime
View raw message