hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Spiegelberg <nspiegelb...@fb.com>
Subject Re: hbase 0.94.0
Date Mon, 30 Jan 2012 19:53:49 GMT
I think it's also good to make a crucial distinction here: it's not like
Facebook has a concrete wide-scale upgrade strategy, these 3 points are
deal-breakers that won't allow us to even entertain an upgrade strategy.
This is just as critical as HDFS data loss was in 0.90: it's something we
can't entertain & is a mandatory hurdle to trunk being considered a
realistic option.  That said, I personally would like to upgrade our
internal version to more closely align with other developers.  It will
reduce our porting overhead and increase the amount of benefit we can give
the community & visa-versa.

On 1/30/12 11:46 AM, "Ted Yu" <yuzhihong@gmail.com> wrote:

>Before long term wire and format compatibility is reached, I am -0 on time
>based release.
>
>As Nicolas pointed out, FB wants to migrate from 0.89-fb to 0.94
>If we do time-based releases without wire and format compatibility, it
>would consume a lot of our energy satisfying such requirements from
>different companies.
>
>I propose wire and format compatibility as the main theme for 0.96
>
>Cheers
>
>On Mon, Jan 30, 2012 at 11:40 AM, Andrew Purtell
><apurtell@apache.org>wrote:
>
>> > Maybe the discussion should be whether 0.94 should be a time-gated or
>> > feature-gated release? IMO we already have enough good stuff in there
>> > on the perf front. It will be great if the checksum improvement makes
>> > it, but if it doesn't, I'd rather have a release than delay for it.
>>
>>
>> Time gated releases makes sense going forward for a few reasons, mark me
>> down for that option.
>>
>>
>> Best regards,
>>
>>     - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>>
>> ----- Original Message -----
>> > From: Todd Lipcon <todd@cloudera.com>
>> > To: dev@hbase.apache.org
>> > Cc: lars hofhansl <lhofhansl@yahoo.com>; Matt Corgan <
>> mcorgan@hotpads.com>
>> > Sent: Monday, January 30, 2012 10:44 AM
>> > Subject: Re: hbase 0.94.0
>> >
>> > On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >>  Initial results from HBASE-5074, support checksums in HBase block
>> cache,
>> >>  look promising.
>> >>
>> >>  Shall we discuss whether 0.94 should include it (assuming Dhruba can
>> finish
>> >>  the feature in Feb.) ?
>> >
>> > If it's a time-based release, then there's nothing to discuss. Either
>> > it's done in time, in which case it's part of 94, or it's not, in
>> > which case it's part of 96, right?
>> >
>> > Maybe the discussion should be whether 0.94 should be a time-gated or
>> > feature-gated release? IMO we already have enough good stuff in there
>> > on the perf front. It will be great if the checksum improvement makes
>> > it, but if it doesn't, I'd rather have a release than delay for it.
>> >
>> > -Todd
>> >
>> >>
>> >>  Cheers
>> >>
>> >>  On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lhofhansl@yahoo.com>
>> > wrote:
>> >>
>> >>>  Salesforce will jumpstart with 0.94. :) Our dev cluster has the
>> current
>> >>>  trunk on it.
>> >>>
>> >>>  OK let's just add these to 0.94:
>> >>>  HBASE-4218
>> >>>
>> >>>  HBASE-4608
>> >>>  HBASE-5128
>> >>>  script necessary to do HBASE-5293 in 0.96
>> >>>
>> >>>
>> >>>  And defer the rest to 0.96, and branch 0.94 soon'ish.
>> >>>  Do you think you can do a 0.92 and trunk version of HBASE-5128?
>> >>>
>> >>>  -- Lars
>> >>>
>> >>>
>> >>>  ________________________________
>> >>>   From: Jonathan Hsieh <jon@cloudera.com>
>> >>>  To: Matt Corgan <mcorgan@hotpads.com>
>> >>>  Cc: lars hofhansl <lhofhansl@yahoo.com>; dev
>> > <dev@hbase.apache.org>
>> >>>  Sent: Friday, January 27, 2012 6:11 PM
>> >>>  Subject: Re: hbase 0.94.0
>> >>>
>> >>>  Lars, Matt,
>> >>>
>> >>>  I like Matt's suggestion -- as long as it is not a burden let's
>> > keep it in.
>> >>>  If it becomes one, let's do what is best for the project.
>> >>>
>> >>>  If Cloudera needs to do the upgrade path, we'll provide one.  If it
>> > doesn't
>> >>>  go to the apache repo, we'll most likely open source it on github
>> > or
>> >>>  something, or "keep it on the jira" like some of the repair
>> > ruby scripts.
>> >>>
>> >>>  Question -- to the Trend/SalesForce/FB folks can you talk about
>>your
>> > plans
>> >>>  for upgrades?  Are you guys moving from 0.89/0.90ish to 0.92
>>already?
>> >  or
>> >>>  are you guys going to have to do the direct upgrade to 0.94 from
>>0.90
>> > land
>> >>>  as well?
>> >>>
>> >>>  Jon.
>> >>>
>> >>>
>> >>>  On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan
>> > <mcorgan@hotpads.com> wrote:
>> >>>
>> >>>  > The reason I brought it up is now that Mikhail committed the data
>> > block
>> >>>  > encoding I was going to take a stab at adding the prefix trie
>> > encoding I
>> >>>  > was working on this past summer.  My plan is to first make a
>> > minimally
>> >>>  > invasive patch to prove correctness.  But, after that there will
>> > probably
>> >>>  > be some big performance gains to be had from reworking some of
>>the
>> > things
>> >>>  > like KeyValueScanner which I would not have the courage to get
>> > working
>> >>>  with
>> >>>  > v1.
>> >>>  >
>> >>>  > So, that was why I asked, but all of that is still more
>> > hypothetical than
>> >>>  > real, and I don't even know if the first part will be done
>> > before
>> >>>  branching
>> >>>  > .94 at the end of February.  Makes sense to me to not delete v1
>> > until
>> >>>  > there's a good reason to, which it doesn't look like we
>> > have yet.  If I
>> >>>  get
>> >>>  > to the point where v1 is halting progress then we can reevaluate
>> > based on
>> >>>  > more specific issues.  Maybe none of the prefix trie will even
>> > make
>> >>>  .94...
>> >>>  >
>> >>>  > ..sent from my phone
>> >>>  > On Jan 27, 2012 1:55 PM, "lars hofhansl"
>> > <lhofhansl@yahoo.com> wrote:
>> >>>  >
>> >>>  >>  Hey Jon,
>> >>>  >>
>> >>>  >> understood. Makes 0.94 hard, though. If we decide now to have
>> > a 0.90 to
>> >>>  >> 0.94 upgrade path and then timing does not work out and nobody
>> > signs up
>> >>>  for
>> >>>  >> the testing because it 0.92 is more convenient we'd have
>> > gone through
>> >>>  this
>> >>>  >> for nothing.
>> >>>  >>
>> >>>  >> So... Thinking about this more I am -1 on supporting an
>> > official upgrade
>> >>>  >> path other then from one release to the next.
>> >>>  >> That said, we do not have to break things intentionally.
>> >>>  >>
>> >>>  >>
>> >>>  >> I'm fine pushing HBASE-2600 and HFile v1 removal out of
>> > 0.94... As long
>> >>>  >> as we won't havethe same argument for 0.96 :)
>> >>>  >>
>> >>>  >> And I am not aware of any file compatibility issues.
>> >>>  >>
>> >>>  >>
>> >>>  >> We can also leave the 0.92 migration code in, but not
>> > officially support
>> >>>  >> it as 0.90 to 0.94 path.
>> >>>  >> Then CDH4 can make sure (if needed) that it is all working.
>> >>>  >>
>> >>>  >>
>> >>>  >> Does that work for you guys?
>> >>>  >>
>> >>>  >> -- Lars
>> >>>  >>
>> >>>  >>
>> >>>  >>
>> >>>  >> ________________________________
>> >>>  >>  From: Jonathan Hsieh <jon@cloudera.com>
>> >>>  >> To: dev@hbase.apache.org; lars hofhansl
>> > <lhofhansl@yahoo.com>
>> >>>  >> Sent: Friday, January 27, 2012 10:10 AM
>> >>>  >> Subject: Re: hbase 0.94.0
>> >>>  >>
>> >>>  >>
>> >>>  >> Lars,
>> >>>  >>
>> >>>  >> The upcoming CDH4 Beta HBase will be based on the latest
>> > hotness,
>> >>>  0.92.0.
>> >>>  >> A CDH4 GA HBase will have to have an upgrade path from CDH3
>> > HBase. If
>> >>>  that
>> >>>  >> HBase is 0.92 based, we'll test that, and if timing works
>> > out and we
>> >>>  decide
>> >>>  >> 0.94, we'll have to have a path (0.90->0.94) for than
>> > and will test
>> >>>  that.
>> >>>  >>
>> >>>  >> HBASE-2600 is a big change of encoding of meta, while my
>> > understanding
>> >>>  is
>> >>>  >> that 0.90->0,92 is a graceful HFile format conversion.
 Are
>> > there are
>> >>>  >> things currently in trunk that further break compatiblity
of
>> > the file
>> >>>  >> format? (what jiras?)
>> >>>  >>
>> >>>  >> Jon.
>> >>>  >>
>> >>>  >>
>> >>>  >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
>> > <lhofhansl@yahoo.com>
>> >>>  >> wrote:
>> >>>  >>
>> >>>  >> Not removing code for upgrade is fine.
>> >>>  >> >
>> >>>  >> >Todd, is Cloudera signing up for testing this path (0.90
>> > to 0.94)?
>> >>>  >> >
>> >>>  >> >
>> >>>  >> >Stack, what's your feeling w.r.t. to HBASE-2600, will
>> > keeping the 0.90
>> >>>  >> -> 0.92 migration path
>> >>>  >> >make the migration code for HBASE-2600 (much) more
>> > complicated in 0.94?
>> >>>  >> >
>> >>>  >> >
>> >>>  >> >-- Lars
>> >>>  >> >
>> >>>  >> >
>> >>>  >> >
>> >>>  >> >----- Original Message -----
>> >>>  >> >From: Stack <stack@duboce.net>
>> >>>  >> >To: dev@hbase.apache.org
>> >>>  >> >Cc:
>> >>>  >> >Sent: Friday, January 27, 2012 9:02 AM
>> >>>  >> >Subject: Re: hbase 0.94.0
>> >>>  >> >
>> >>>  >> >
>> >>>  >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack
>> > <stack@duboce.net> wrote:
>> >>>  >> >> In this particular case, there was no explicit
>> > migration step needed
>> >>>  >> >> going 0.90 to 0.92.  Upgrading from 0.90 to 0.94
>> > should just be
>> >>>  >> >> running the 0.92 to 0.94 migration script (if there
>> > is one).
>> >>>  >> >>
>> >>>  >> >
>> >>>  >> >Thinking more, the above only really holds if we keep
the
>> > .META.
>> >>>  >> >migration code that runs in 0.92 on startup on into 0.94
>> > (I have a
>> >>>  >> >patch here where I'm removing it... I should put this
>> > bit of code
>> >>>  >> >back).
>> >>>  >> >
>> >>>  >> >I see Todd that you vote against removing hfile v1 in
>> > 0.94.  To make
>> >>>  >> >going from CDH3 to CDH4, we should not purge any migrating
>> > code or old
>> >>>  >> >version support.
>> >>>  >> >
>> >>>  >> >I'd be fine w/ that.  We'll need to work hard to
>> > ensure it taking it
>> >>>  >> >on as a principal for 0.94.  Ok w/ you "Iron
>> > Hand" Lars?
>> >>>  >> >
>> >>>  >> >St.Ack
>> >>>  >> >
>> >>>  >> >
>> >>>  >>
>> >>>  >>
>> >>>  >> --
>> >>>  >> // Jonathan Hsieh (shay)
>> >>>  >> // Software Engineer, Cloudera
>> >>>  >>
>> >>>  >> // jon@cloudera.com
>> >>>  >>
>> >>>  >
>> >>>
>> >>>
>> >>>  --
>> >>>  // Jonathan Hsieh (shay)
>> >>>  // Software Engineer, Cloudera
>> >>>  // jon@cloudera.com
>> >>>
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>


Mime
View raw message