hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Demai Ni <nid...@gmail.com>
Subject Re: HBase Hackathon @ Salesforce 05/06/2014 notes
Date Mon, 12 May 2014 21:17:24 GMT
Stack,

thanks for sharing.

I can't access the google doc mentioned after "...Ryan then presented this
doc..."  due to permission. Is there a way to get it? thanks

Demai


On Mon, May 12, 2014 at 12:12 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> Nice! Thanks for sharing St.Ack.
>
> JM
>
>
> 2014-05-12 13:32 GMT-04:00 Stack <stack@duboce.net>:
>
> > Below are some rough notes taken during first 20 minutes of our
> > hackathon/dev meetup last Tuesday (Your secretary had to abandon
> > note-taking for white-boarding duty and failed to pass the baton).
> >
> > There were about ~50 folks in attendance with representatives
> > from a variety of organizations.
> >
> > We started by promoting the items listed on the hackathon
> > meetup page to the white board:
> >
> > http://files.meetup.com/1350316/IMG_20140506_190458.jpg
> >
> > The list of topics up for discussions were:
> >
> > + Agreeing on types among hbase-related projects (phoenix,
> > kite, etc.).
> > + Discussion around colocation of hbase master and meta
> > and the master also doing regionserver services and whether
> > we should retain the current topology (no regions on the
> > master or backup master) for 1.0.
> > + The ongoing 'consensus' work (putting zk usage behind
> > a pluggable interface).
> > + Transactions over HBase (Continuuity work, Xiaomi
> > percolator, and VCNC's work).
> > + Netty the server
> > + HTrace
> > + Multitenancy
> >
> > We started talking types.  It took a while to get going . Below are the
> raw
> > notes:
> >
> > Static, dynamic or what?
> >
> > Lars Hofhansl: Is hbase a byte store? Should types be in there? If hbase
> > knew
> > about the type then could we exploit it internally.
> >
> > Should HBase know about types? Thought is that HBase could do some
> > perf stuff if it was cognizant of types.  Maybe later do this or
> > start out w/ a few common types first.
> >
> > At least one encoding strategy, work on this first.
> >
> > Make it so we don't design ourselves into a corner.
> >
> > Talking to Enis, Hive & Pig... what do these projects want?  Plug in
> codec?
> >
> > Kite project, tries to do a layer above byte store rather than in the fs.
> >
> > Encodings are not the same across storage engines for Kite.
> >
> > Does Parquet need to do float or integer?  But that is where it gets its
> > perf advantage.
> >
> > Parquet does not necessarily work inside hbase.
> >
> > Kite as a lib?  That phoenix could use?
> >
> > Could add a phoenix encoding to Kite?  Yes.  But lets agree and then
> retire
> > the avro encoding.
> >
> > Encoding is separate from schema.
> >
> > Ryan Blue: Don't want to say phoenix is on kite.  Just want to focus on
> > encoding.
> >
> > James Taylor: Phoenix should be under kite, not on top of kite.
> >
> > Ryan: move to common encoding, and a lib to serialize....
> >
> > Jon Hsieh: where this could be used... phoenix encoding for kite... then
> > all
> > could use it writing out.
> >
> > Lars: What you doing at HP for type encoding.
> >
> > HP: Our code is in C.  All byte arrays.  We have a way of doing floats,
> > etc.
> >
> > Ryan: But you want a serializing too?
> >
> > Lars: How we start in on this?
> >
> > Ryan: I sent out a doc. Ryan then presented this doc:
> >
> >
> >
> https://docs.google.com/a/cloudera.com/document/d/15INOaxyifycpFxvB6xNdoj96JbOpslEywjkaoG8MURE/edit#heading=h.o1cgqtsqgqyg
> >
> > Pick just a few simple types and get agreement on these first?
> >
> > HP: We have seen in the past that this works for a while but then folks
> > figure
> > out that their complex types are slower than they should be and they
> start
> > coming up w/ their own encodings as workarounds; now you are back to
> > square one.... you need to version it all.
> >
> > Was proposed that we try and unify around a typing strategy that would
> work
> > as
> > the ONE way to do it in HBase with Phoenix going to try and come up on it
> > first.
> > To be continued up on the HBase dev list.
> >
> > Another proposal that had alot of nods in favor was that we not require
> > 1.0 and 2.0 to be compatible.
> >
> > Notes ran out at this stage.
> >
> > St.Ack
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message