hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: HBase Hackathon @ Salesforce 05/06/2014 notes
Date Mon, 12 May 2014 19:12:19 GMT
Nice! Thanks for sharing St.Ack.

JM


2014-05-12 13:32 GMT-04:00 Stack <stack@duboce.net>:

> Below are some rough notes taken during first 20 minutes of our
> hackathon/dev meetup last Tuesday (Your secretary had to abandon
> note-taking for white-boarding duty and failed to pass the baton).
>
> There were about ~50 folks in attendance with representatives
> from a variety of organizations.
>
> We started by promoting the items listed on the hackathon
> meetup page to the white board:
>
> http://files.meetup.com/1350316/IMG_20140506_190458.jpg
>
> The list of topics up for discussions were:
>
> + Agreeing on types among hbase-related projects (phoenix,
> kite, etc.).
> + Discussion around colocation of hbase master and meta
> and the master also doing regionserver services and whether
> we should retain the current topology (no regions on the
> master or backup master) for 1.0.
> + The ongoing 'consensus' work (putting zk usage behind
> a pluggable interface).
> + Transactions over HBase (Continuuity work, Xiaomi
> percolator, and VCNC's work).
> + Netty the server
> + HTrace
> + Multitenancy
>
> We started talking types.  It took a while to get going . Below are the raw
> notes:
>
> Static, dynamic or what?
>
> Lars Hofhansl: Is hbase a byte store? Should types be in there? If hbase
> knew
> about the type then could we exploit it internally.
>
> Should HBase know about types? Thought is that HBase could do some
> perf stuff if it was cognizant of types.  Maybe later do this or
> start out w/ a few common types first.
>
> At least one encoding strategy, work on this first.
>
> Make it so we don't design ourselves into a corner.
>
> Talking to Enis, Hive & Pig... what do these projects want?  Plug in codec?
>
> Kite project, tries to do a layer above byte store rather than in the fs.
>
> Encodings are not the same across storage engines for Kite.
>
> Does Parquet need to do float or integer?  But that is where it gets its
> perf advantage.
>
> Parquet does not necessarily work inside hbase.
>
> Kite as a lib?  That phoenix could use?
>
> Could add a phoenix encoding to Kite?  Yes.  But lets agree and then retire
> the avro encoding.
>
> Encoding is separate from schema.
>
> Ryan Blue: Don't want to say phoenix is on kite.  Just want to focus on
> encoding.
>
> James Taylor: Phoenix should be under kite, not on top of kite.
>
> Ryan: move to common encoding, and a lib to serialize....
>
> Jon Hsieh: where this could be used... phoenix encoding for kite... then
> all
> could use it writing out.
>
> Lars: What you doing at HP for type encoding.
>
> HP: Our code is in C.  All byte arrays.  We have a way of doing floats,
> etc.
>
> Ryan: But you want a serializing too?
>
> Lars: How we start in on this?
>
> Ryan: I sent out a doc. Ryan then presented this doc:
>
>
> https://docs.google.com/a/cloudera.com/document/d/15INOaxyifycpFxvB6xNdoj96JbOpslEywjkaoG8MURE/edit#heading=h.o1cgqtsqgqyg
>
> Pick just a few simple types and get agreement on these first?
>
> HP: We have seen in the past that this works for a while but then folks
> figure
> out that their complex types are slower than they should be and they start
> coming up w/ their own encodings as workarounds; now you are back to
> square one.... you need to version it all.
>
> Was proposed that we try and unify around a typing strategy that would work
> as
> the ONE way to do it in HBase with Phoenix going to try and come up on it
> first.
> To be continued up on the HBase dev list.
>
> Another proposal that had alot of nods in favor was that we not require
> 1.0 and 2.0 to be compatible.
>
> Notes ran out at this stage.
>
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message