hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Minutes from Developer Meetup at HWX October 24th
Date Tue, 05 Nov 2013 05:47:31 GMT
On Mon, Nov 4, 2013 at 9:13 PM, Andrew Purtell <apurtell@apache.org> wrote:

> When last we spoke on dev@ about 0.98 I suggested branching one month
> after
> the 0.96.0 release. There was no comment or objection. That would be late
> November then. No problem to branch sooner.
>
> Sorry none of us at Intel could be there in person. For 0.98, to add to the
> below list there are a couple of security related issues we're aiming for
> the release, based on tags, coprocessors, and HFile v3: cell ACLs,
> visibility labels, and transparent encryption.
>
>
Branch end of Nov. sounds about right for getting the above in.
Good on you Andrew,
St.Ack


On Mon, Nov 4, 2013 at 9:13 PM, Andrew Purtell <apurtell@apache.org> wrote:

> When last we spoke on dev@ about 0.98 I suggested branching one month
> after
> the 0.96.0 release. There was no comment or objection. That would be late
> November then. No problem to branch sooner.
>
> Sorry none of us at Intel could be there in person. For 0.98, to add to the
> below list there are a couple of security related issues we're aiming for
> the release, based on tags, coprocessors, and HFile v3: cell ACLs,
> visibility labels, and transparent encryption.
>
>
>
> On Saturday, November 2, 2013, Stack wrote:
>
> > Below are rough minutes from the developer meetup last Thursday week down
> > at HortonWorks: http://www.meetup.com/hackathon/events/144366512/
> >
> > The agenda was fast moving and the notes I kept were sparse (pardon me).
> >  Hopefully the below at least conveys some flavor of what transpired.
> >
> >
> > Below is proposed agenda with discussion filled in in between in italics.
> >
> > Git/Gerrit<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.bi2n1jnf8wr9
> > >
> >
> > Lieutenants<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ts3zsiollp5s
> > >
> >
> > MTTR<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2
> > >
> >
> > Distributed log
> > replay<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.5nxjf2azyr63
> > >
> >
> > Region online for
> > writes<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.7h4u62q0yu1t
> > >
> >
> > Client<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.dxyzoaydy7up
> > >
> >
> > 0.98<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.jny6o6f3j3x8
> > >
> >
> > 1.0<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.4sjuoukolha1
> > >
> >
> > Release passes IT tests for time period before
> > release?<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ijcofohq0rk1
> > >
> >
> > Sequenceid<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.14eyjkb3glf1
> > >
> >
> > MVCC<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.hbv260gur4ui
> > >
> >
> > Multiwal<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.tab1edfjmpgq
> > >
> >
> > Speculative Read<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.3xr7mrnpmpby
> > >
> >
> > Favored node finish
> > up<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.2s9y1srcix0z
> > >
> >
> > Compactions<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.23pauxcj0ux8
> > >
> >
> > Stripe?<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.xsncn8yoed07
> > >
> >
> > Master Redo<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2
> > >
> >
> > Secondary indices from
> > phoenix<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.mk56oao2p3vj
> > >
> > 
> >
> > Git/Gerrit
> >
> > Move to this? Or do Git without gerrit ?
> >
> > All were +1 on going to GIT.  A gentleman Accumulo fellow present
> > volunteered to pass us his script for how they made the transition.
> > DISCUSSION already started on dev list.
> >
> > Lieutenants/Component Owners
> >
> > Review.  How is this going?  What can we do to improve?  Not enough
> reviews
> > going on.
> >
> > The list of Component Owners needs an edit.  It is stale.  We need to
> > revive/renew/refresh this initiative.
> >
> > More Friction Committing
> >
> > Not enough reviews of stuff going in and this is a db after all? +1
> > (eclark) +1 from Dave
> >
> > Generally agreed.  Pointed at Lieutenants/Component Owners role. Was
> > thought there should be some sort of automated performance test as part
> of
> > hadoopqa.  A benchmark proving no degradation of a claimed improvement is
> > required.
> >
> > Better was adding some set of general micro benchmarks... and then do
> them
> > on your machine before and after and paste findings.
> >
> > Compat testing
> >
> > Now that 0.96.0 is out the door, we should be careful about changes that
> > are compat-breaking.  Any ideas on how to best avoid/detect such changes?
> >  Also maybe have a discussion about how wire compat/protobufs changes how
> > we do things, how to best utilize it, pitfalls to avoid, what it can’t
> do?
> >
> > Formalize compat matrix? Go over the matrix we had 1 year ago. Do we want
> > to support all combinations, or only between two major releases with some
> > rolling upgrade model? How to avoid current situation where client does
> not
> > scale -- found at last minute running on 80 node cluster!
> >
> > Test framework do this?  But would have to be a non-apache context
> because
> > no resources there to do it.  There are the jdiff and compare. We have to
> > do rolling upgrade tests. Deprecations in .protos too. Remove after a
> > release. Just go version numbers. Write up the matrix. How to avoid
> issues
> > like the recent client not scaling.  Need perf tests.  Need
> > microbenchmarks. Modular so can pull out and simulate rpc.  Tests like
> > that.  Standup simple server done in proto
> > MTTRDistributed log replay
> >
> > Relax semantics; it is ok to allow out-of-order edits?  Or fix?
> >
> > Distributed log replay going to be done in 0.98 using tags.
> > Region online for writes
> >
> > Lets just do this.
> >
> > Client
> >
> > - Short term perf/scalability regression (0.96) time frame
> >
> > - Long term (0.98/1.x/2.x) time frame.
> >
> > - Asynchbase on 0.96 - almost there.
> >
> > Should we remove support for setting timestamp. Lars says set it in
> > HTableDescription. Sequence number and ts.  Sequenceid  Perhaps add new
> > coordinate. Out-of-order deletes. Ts w/ the value.  Make it configurable
> > and do optimization.
> >
> > *Aditya brought up new c-client effort.*
> > 0.98
> >
> > - tags.
> >
> > - reverse scan
> >
> > - issue with cells? (intel guys at mtg?)
> >
> > Where is the branch?
> > 1.0
> >
> > is 0.98 == 1.0?
> >
> > - when to drop hadoop1 support? next year?
> >
> >
> > 98.x becomes 1.0 Security not as a CP but in codepath. Do check in the
> > code. Make security first class. But security was always good for dev'ing
> > CP. Make it a required CP. Integrated w/ simple testing. Enable
> > Authorization by default.  Permissive mode needs to be added. Small perf
> > price.
> >
> >
> > We need better defaults -- especially for 1.0. Revisit the docs, the
> > refguide.  An edit. Questions, the docbook.... Formatting is a pain.
> > Docathon?
> >  ….but no one would show up. Drop hadoop1 post 1.0.
> >
> > New release adoption
> >
> > - What can we do for 0.96 adoption
> >
> > - People are still using 0.90
> >
> > Within versions, update... You are in the middle of a rolling upgrade.
> Can
> > we make it easier for folks to update? Advertising that upgrade to 0.96
> is
> > successful. Rolling upgrade needs to be in the master, not out in a
> script.
> > Release passes IT tests for time period before release?
> >
> > 48 hours  Real honest, regionserver, datanodes, zookeepers all being
> > killed. Committers releasing it for a time.  Failing a 48 hour test is
> > enough to sink an RC. We have a bar and we should keep raising it.
> >
> > Elliott will put up bar for test release.
> >
> > SequenceidMVCC
> >
> > Unify sequenceid and mvcc
> >
> > Do we need locks at all?
> > MultiwalSpeculative Read
> >
> > Raise awareness with hdfs brothers and sisters
> > Favored node finish up
> >
> > Add consideration to the balancer
> >
> > Need to fix balancer. Not AM job.  Its in AM and in Balancer.
> >
> >
> > CompactionsStripe?
> >
> > Get some empirical data to see if different types of compactions are
> > better.
> > Master Redo
> >
> > Jimmy says evolve instead of rewrite
> > Secondary indices from phoenix?
> >
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message