hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Corgan <mcor...@hotpads.com>
Subject Re: hbase 0.94.0
Date Mon, 30 Jan 2012 20:01:23 GMT
I had assumed facebook had integrated HFileV2 into their 0.89 branch a
while back and that pretty much all HFiles would have been converted to v2
by now.  I guess that is not correct...


On Mon, Jan 30, 2012 at 11:53 AM, Nicolas Spiegelberg
<nspiegelberg@fb.com>wrote:

> I think it's also good to make a crucial distinction here: it's not like
> Facebook has a concrete wide-scale upgrade strategy, these 3 points are
> deal-breakers that won't allow us to even entertain an upgrade strategy.
> This is just as critical as HDFS data loss was in 0.90: it's something we
> can't entertain & is a mandatory hurdle to trunk being considered a
> realistic option.  That said, I personally would like to upgrade our
> internal version to more closely align with other developers.  It will
> reduce our porting overhead and increase the amount of benefit we can give
> the community & visa-versa.
>
> On 1/30/12 11:46 AM, "Ted Yu" <yuzhihong@gmail.com> wrote:
>
> >Before long term wire and format compatibility is reached, I am -0 on time
> >based release.
> >
> >As Nicolas pointed out, FB wants to migrate from 0.89-fb to 0.94
> >If we do time-based releases without wire and format compatibility, it
> >would consume a lot of our energy satisfying such requirements from
> >different companies.
> >
> >I propose wire and format compatibility as the main theme for 0.96
> >
> >Cheers
> >
> >On Mon, Jan 30, 2012 at 11:40 AM, Andrew Purtell
> ><apurtell@apache.org>wrote:
> >
> >> > Maybe the discussion should be whether 0.94 should be a time-gated or
> >> > feature-gated release? IMO we already have enough good stuff in there
> >> > on the perf front. It will be great if the checksum improvement makes
> >> > it, but if it doesn't, I'd rather have a release than delay for it.
> >>
> >>
> >> Time gated releases makes sense going forward for a few reasons, mark me
> >> down for that option.
> >>
> >>
> >> Best regards,
> >>
> >>     - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >>
> >> ----- Original Message -----
> >> > From: Todd Lipcon <todd@cloudera.com>
> >> > To: dev@hbase.apache.org
> >> > Cc: lars hofhansl <lhofhansl@yahoo.com>; Matt Corgan <
> >> mcorgan@hotpads.com>
> >> > Sent: Monday, January 30, 2012 10:44 AM
> >> > Subject: Re: hbase 0.94.0
> >> >
> >> > On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> >>  Initial results from HBASE-5074, support checksums in HBase block
> >> cache,
> >> >>  look promising.
> >> >>
> >> >>  Shall we discuss whether 0.94 should include it (assuming Dhruba can
> >> finish
> >> >>  the feature in Feb.) ?
> >> >
> >> > If it's a time-based release, then there's nothing to discuss. Either
> >> > it's done in time, in which case it's part of 94, or it's not, in
> >> > which case it's part of 96, right?
> >> >
> >> > Maybe the discussion should be whether 0.94 should be a time-gated or
> >> > feature-gated release? IMO we already have enough good stuff in there
> >> > on the perf front. It will be great if the checksum improvement makes
> >> > it, but if it doesn't, I'd rather have a release than delay for it.
> >> >
> >> > -Todd
> >> >
> >> >>
> >> >>  Cheers
> >> >>
> >> >>  On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lhofhansl@yahoo.com
> >
> >> > wrote:
> >> >>
> >> >>>  Salesforce will jumpstart with 0.94. :) Our dev cluster has the
> >> current
> >> >>>  trunk on it.
> >> >>>
> >> >>>  OK let's just add these to 0.94:
> >> >>>  HBASE-4218
> >> >>>
> >> >>>  HBASE-4608
> >> >>>  HBASE-5128
> >> >>>  script necessary to do HBASE-5293 in 0.96
> >> >>>
> >> >>>
> >> >>>  And defer the rest to 0.96, and branch 0.94 soon'ish.
> >> >>>  Do you think you can do a 0.92 and trunk version of HBASE-5128?
> >> >>>
> >> >>>  -- Lars
> >> >>>
> >> >>>
> >> >>>  ________________________________
> >> >>>   From: Jonathan Hsieh <jon@cloudera.com>
> >> >>>  To: Matt Corgan <mcorgan@hotpads.com>
> >> >>>  Cc: lars hofhansl <lhofhansl@yahoo.com>; dev
> >> > <dev@hbase.apache.org>
> >> >>>  Sent: Friday, January 27, 2012 6:11 PM
> >> >>>  Subject: Re: hbase 0.94.0
> >> >>>
> >> >>>  Lars, Matt,
> >> >>>
> >> >>>  I like Matt's suggestion -- as long as it is not a burden let's
> >> > keep it in.
> >> >>>  If it becomes one, let's do what is best for the project.
> >> >>>
> >> >>>  If Cloudera needs to do the upgrade path, we'll provide one. 
If it
> >> > doesn't
> >> >>>  go to the apache repo, we'll most likely open source it on github
> >> > or
> >> >>>  something, or "keep it on the jira" like some of the repair
> >> > ruby scripts.
> >> >>>
> >> >>>  Question -- to the Trend/SalesForce/FB folks can you talk about
> >>your
> >> > plans
> >> >>>  for upgrades?  Are you guys moving from 0.89/0.90ish to 0.92
> >>already?
> >> >  or
> >> >>>  are you guys going to have to do the direct upgrade to 0.94 from
> >>0.90
> >> > land
> >> >>>  as well?
> >> >>>
> >> >>>  Jon.
> >> >>>
> >> >>>
> >> >>>  On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan
> >> > <mcorgan@hotpads.com> wrote:
> >> >>>
> >> >>>  > The reason I brought it up is now that Mikhail committed
the data
> >> > block
> >> >>>  > encoding I was going to take a stab at adding the prefix
trie
> >> > encoding I
> >> >>>  > was working on this past summer.  My plan is to first make
a
> >> > minimally
> >> >>>  > invasive patch to prove correctness.  But, after that there
will
> >> > probably
> >> >>>  > be some big performance gains to be had from reworking some
of
> >>the
> >> > things
> >> >>>  > like KeyValueScanner which I would not have the courage to
get
> >> > working
> >> >>>  with
> >> >>>  > v1.
> >> >>>  >
> >> >>>  > So, that was why I asked, but all of that is still more
> >> > hypothetical than
> >> >>>  > real, and I don't even know if the first part will be done
> >> > before
> >> >>>  branching
> >> >>>  > .94 at the end of February.  Makes sense to me to not delete
v1
> >> > until
> >> >>>  > there's a good reason to, which it doesn't look like we
> >> > have yet.  If I
> >> >>>  get
> >> >>>  > to the point where v1 is halting progress then we can reevaluate
> >> > based on
> >> >>>  > more specific issues.  Maybe none of the prefix trie will
even
> >> > make
> >> >>>  .94...
> >> >>>  >
> >> >>>  > ..sent from my phone
> >> >>>  > On Jan 27, 2012 1:55 PM, "lars hofhansl"
> >> > <lhofhansl@yahoo.com> wrote:
> >> >>>  >
> >> >>>  >>  Hey Jon,
> >> >>>  >>
> >> >>>  >> understood. Makes 0.94 hard, though. If we decide now
to have
> >> > a 0.90 to
> >> >>>  >> 0.94 upgrade path and then timing does not work out and
nobody
> >> > signs up
> >> >>>  for
> >> >>>  >> the testing because it 0.92 is more convenient we'd have
> >> > gone through
> >> >>>  this
> >> >>>  >> for nothing.
> >> >>>  >>
> >> >>>  >> So... Thinking about this more I am -1 on supporting
an
> >> > official upgrade
> >> >>>  >> path other then from one release to the next.
> >> >>>  >> That said, we do not have to break things intentionally.
> >> >>>  >>
> >> >>>  >>
> >> >>>  >> I'm fine pushing HBASE-2600 and HFile v1 removal out
of
> >> > 0.94... As long
> >> >>>  >> as we won't havethe same argument for 0.96 :)
> >> >>>  >>
> >> >>>  >> And I am not aware of any file compatibility issues.
> >> >>>  >>
> >> >>>  >>
> >> >>>  >> We can also leave the 0.92 migration code in, but not
> >> > officially support
> >> >>>  >> it as 0.90 to 0.94 path.
> >> >>>  >> Then CDH4 can make sure (if needed) that it is all working.
> >> >>>  >>
> >> >>>  >>
> >> >>>  >> Does that work for you guys?
> >> >>>  >>
> >> >>>  >> -- Lars
> >> >>>  >>
> >> >>>  >>
> >> >>>  >>
> >> >>>  >> ________________________________
> >> >>>  >>  From: Jonathan Hsieh <jon@cloudera.com>
> >> >>>  >> To: dev@hbase.apache.org; lars hofhansl
> >> > <lhofhansl@yahoo.com>
> >> >>>  >> Sent: Friday, January 27, 2012 10:10 AM
> >> >>>  >> Subject: Re: hbase 0.94.0
> >> >>>  >>
> >> >>>  >>
> >> >>>  >> Lars,
> >> >>>  >>
> >> >>>  >> The upcoming CDH4 Beta HBase will be based on the latest
> >> > hotness,
> >> >>>  0.92.0.
> >> >>>  >> A CDH4 GA HBase will have to have an upgrade path from
CDH3
> >> > HBase. If
> >> >>>  that
> >> >>>  >> HBase is 0.92 based, we'll test that, and if timing works
> >> > out and we
> >> >>>  decide
> >> >>>  >> 0.94, we'll have to have a path (0.90->0.94) for than
> >> > and will test
> >> >>>  that.
> >> >>>  >>
> >> >>>  >> HBASE-2600 is a big change of encoding of meta, while
my
> >> > understanding
> >> >>>  is
> >> >>>  >> that 0.90->0,92 is a graceful HFile format conversion.
 Are
> >> > there are
> >> >>>  >> things currently in trunk that further break compatiblity
of
> >> > the file
> >> >>>  >> format? (what jiras?)
> >> >>>  >>
> >> >>>  >> Jon.
> >> >>>  >>
> >> >>>  >>
> >> >>>  >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
> >> > <lhofhansl@yahoo.com>
> >> >>>  >> wrote:
> >> >>>  >>
> >> >>>  >> Not removing code for upgrade is fine.
> >> >>>  >> >
> >> >>>  >> >Todd, is Cloudera signing up for testing this path
(0.90
> >> > to 0.94)?
> >> >>>  >> >
> >> >>>  >> >
> >> >>>  >> >Stack, what's your feeling w.r.t. to HBASE-2600,
will
> >> > keeping the 0.90
> >> >>>  >> -> 0.92 migration path
> >> >>>  >> >make the migration code for HBASE-2600 (much) more
> >> > complicated in 0.94?
> >> >>>  >> >
> >> >>>  >> >
> >> >>>  >> >-- Lars
> >> >>>  >> >
> >> >>>  >> >
> >> >>>  >> >
> >> >>>  >> >----- Original Message -----
> >> >>>  >> >From: Stack <stack@duboce.net>
> >> >>>  >> >To: dev@hbase.apache.org
> >> >>>  >> >Cc:
> >> >>>  >> >Sent: Friday, January 27, 2012 9:02 AM
> >> >>>  >> >Subject: Re: hbase 0.94.0
> >> >>>  >> >
> >> >>>  >> >
> >> >>>  >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack
> >> > <stack@duboce.net> wrote:
> >> >>>  >> >> In this particular case, there was no explicit
> >> > migration step needed
> >> >>>  >> >> going 0.90 to 0.92.  Upgrading from 0.90 to
0.94
> >> > should just be
> >> >>>  >> >> running the 0.92 to 0.94 migration script (if
there
> >> > is one).
> >> >>>  >> >>
> >> >>>  >> >
> >> >>>  >> >Thinking more, the above only really holds if we
keep the
> >> > .META.
> >> >>>  >> >migration code that runs in 0.92 on startup on into
0.94
> >> > (I have a
> >> >>>  >> >patch here where I'm removing it... I should put
this
> >> > bit of code
> >> >>>  >> >back).
> >> >>>  >> >
> >> >>>  >> >I see Todd that you vote against removing hfile v1
in
> >> > 0.94.  To make
> >> >>>  >> >going from CDH3 to CDH4, we should not purge any
migrating
> >> > code or old
> >> >>>  >> >version support.
> >> >>>  >> >
> >> >>>  >> >I'd be fine w/ that.  We'll need to work hard to
> >> > ensure it taking it
> >> >>>  >> >on as a principal for 0.94.  Ok w/ you "Iron
> >> > Hand" Lars?
> >> >>>  >> >
> >> >>>  >> >St.Ack
> >> >>>  >> >
> >> >>>  >> >
> >> >>>  >>
> >> >>>  >>
> >> >>>  >> --
> >> >>>  >> // Jonathan Hsieh (shay)
> >> >>>  >> // Software Engineer, Cloudera
> >> >>>  >>
> >> >>>  >> // jon@cloudera.com
> >> >>>  >>
> >> >>>  >
> >> >>>
> >> >>>
> >> >>>  --
> >> >>>  // Jonathan Hsieh (shay)
> >> >>>  // Software Engineer, Cloudera
> >> >>>  // jon@cloudera.com
> >> >>>
> >> >
> >> >
> >> >
> >> > --
> >> > Todd Lipcon
> >> > Software Engineer, Cloudera
> >> >
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message