hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: hbase 0.94.0
Date Mon, 30 Jan 2012 20:01:25 GMT
> From: Nicolas Spiegelberg <nspiegelberg@fb.com>
> 2) Client/Server compatibility.  Even if we upgraded the servers to

> 94/96/whatever, we still will have a lot of clients running 89 & we 
> can't


0.92+ supports pluggable RPC engines. Can you manage 0.89 compat that way?


> 3) Durable Server Upgrade.  It does bother me that people are very quick
> to consider removing both RPC & persistent data compatibility from 90 to
> trunk (a branch that we're still actively releasing minor upgrades for).
> We will need the ability to mutate the HDFS & ZK persistent data from the
> 89 format to trunk.  Specifically, work like HFileV1 reader is critical
> for analysis if the upgrade script fails and we need to debug.

Traditionally there have been no guarantees of cross major version compatibility. RPC especially.
Never a rolling upgrade from an 0.90 to an 0.92, for example. For persistent data, there is
a migration utility for upping from one major release to the next.

Regarding RPC, this state of affairs is not really acceptable to anyone any more. Over in
core there's work to move 99% of RPC to protobufs, with only the thinnest Writable header.
In this thread there seem several here who want to tackle this for HBase now.

Regarding major version data migrations, the attitude I believe is pre 1.0 we can entertain
design changes that break compatibility, in search of something that works well enough to
be 1.0. From that point forward, compatibility is a requirement.

In practical terms, with long term scale deployments with compatibility requirements, I think
we are quite near 1.0. Earlier I argued that we should adopt the label ("1.0") like Hadoop
core did in concession to the realities of large scale production deploys, but there were
good counter arguments against doing so, namely that HBase has a few rough edges yet.

> Instead of discussing what classes we can throw away, can we instead think
> about a strategy for long-term, cross-version compatibility that minimally
> hurts trunk development?

+1

Best regards,

    - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) 


----- Original Message -----
> From: Nicolas Spiegelberg <nspiegelberg@fb.com>
> To: "dev@hbase.apache.org" <dev@hbase.apache.org>
> Cc: 
> Sent: Monday, January 30, 2012 11:21 AM
> Subject: Re: hbase 0.94.0
> 
> Currently, we at Facebook are developing mostly on 89 but also doing some
> significant exploratory work on trunk.  I think that most of our
> development will continue to be done on 89 in the near future.  We plan to
> release some lower-risk projects on 94.  However, we won't even entertain
> a wide-scale upgrade strategy until:
> 
> 1) The trunk has more master work.  Specifically 89-master related
> features that we implemented for our grid architecture.  It sounds like
> most of those features are rising in prominence in the Open Source
> community as well (Ted from eBay & Todd from Cloudera, in particular).
> Hopefully, we can collaborate on porting/implementation.
> 2) Client/Server compatibility.  Even if we upgraded the servers to
> 94/96/whatever, we still will have a lot of clients running 89 & we 
> can't
> simply stop/start all of them at the same time.  We need 89/trunk RPC
> compat for client/server.
> 3) Durable Server Upgrade.  It does bother me that people are very quick
> to consider removing both RPC & persistent data compatibility from 90 to
> trunk (a branch that we're still actively releasing minor upgrades for).
> We will need the ability to mutate the HDFS & ZK persistent data from the
> 89 format to trunk.  Specifically, work like HFileV1 reader is critical
> for analysis if the upgrade script fails and we need to debug.
> 
> Instead of discussing what classes we can throw away, can we instead think
> about a strategy for long-term, cross-version compatibility that minimally
> hurts trunk development?  Is HFileV1's presence severely hindering another
> feature?
> 
> On 1/27/12 6:11 PM, "Jonathan Hsieh" <jon@cloudera.com> wrote:
> 
>> Lars, Matt,
>> 
>> I like Matt's suggestion -- as long as it is not a burden let's keep 
> it
>> in.
>>  If it becomes one, let's do what is best for the project.
>> 
>> If Cloudera needs to do the upgrade path, we'll provide one.  If it
>> doesn't
>> go to the apache repo, we'll most likely open source it on github or
>> something, or "keep it on the jira" like some of the repair ruby 
> scripts.
>> 
>> Question -- to the Trend/SalesForce/FB folks can you talk about your plans
>> for upgrades?  Are you guys moving from 0.89/0.90ish to 0.92 already?  or
>> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
>> as well?
>> 
>> Jon.
>> 
>> 
>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mcorgan@hotpads.com> 
> wrote:
>> 
>>>  The reason I brought it up is now that Mikhail committed the data block
>>>  encoding I was going to take a stab at adding the prefix trie encoding 
> I
>>>  was working on this past summer.  My plan is to first make a minimally
>>>  invasive patch to prove correctness.  But, after that there will
>>> probably
>>>  be some big performance gains to be had from reworking some of the
>>> things
>>>  like KeyValueScanner which I would not have the courage to get working
>>> with
>>>  v1.
>>> 
>>>  So, that was why I asked, but all of that is still more hypothetical
>>> than
>>>  real, and I don't even know if the first part will be done before
>>> branching
>>>  .94 at the end of February.  Makes sense to me to not delete v1 until
>>>  there's a good reason to, which it doesn't look like we have 
> yet.  If I
>>> get
>>>  to the point where v1 is halting progress then we can reevaluate based
>>> on
>>>  more specific issues.  Maybe none of the prefix trie will even make
>>> .94...
>>> 
>>>  ..sent from my phone
>>>  On Jan 27, 2012 1:55 PM, "lars hofhansl" 
> <lhofhansl@yahoo.com> wrote:
>>> 
>>>>   Hey Jon,
>>>> 
>>>>  understood. Makes 0.94 hard, though. If we decide now to have a 
> 0.90 to
>>>>  0.94 upgrade path and then timing does not work out and nobody 
> signs
>>>> up for
>>>>  the testing because it 0.92 is more convenient we'd have gone 
> through
>>>> this
>>>>  for nothing.
>>>> 
>>>>  So... Thinking about this more I am -1 on supporting an official
>>>> upgrade
>>>>  path other then from one release to the next.
>>>>  That said, we do not have to break things intentionally.
>>>> 
>>>> 
>>>>  I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... 
> As long
>>>>  as we won't havethe same argument for 0.96 :)
>>>> 
>>>>  And I am not aware of any file compatibility issues.
>>>> 
>>>> 
>>>>  We can also leave the 0.92 migration code in, but not officially
>>>> support
>>>>  it as 0.90 to 0.94 path.
>>>>  Then CDH4 can make sure (if needed) that it is all working.
>>>> 
>>>> 
>>>>  Does that work for you guys?
>>>> 
>>>>  -- Lars
>>>> 
>>>> 
>>>> 
>>>>  ________________________________
>>>>   From: Jonathan Hsieh <jon@cloudera.com>
>>>>  To: dev@hbase.apache.org; lars hofhansl <lhofhansl@yahoo.com>
>>>>  Sent: Friday, January 27, 2012 10:10 AM
>>>>  Subject: Re: hbase 0.94.0
>>>> 
>>>> 
>>>>  Lars,
>>>> 
>>>>  The upcoming CDH4 Beta HBase will be based on the latest hotness,
>>>> 0.92.0.
>>>>  A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. 
> If
>>>> that
>>>>  HBase is 0.92 based, we'll test that, and if timing works out 
> and we
>>>> decide
>>>>  0.94, we'll have to have a path (0.90->0.94) for than and 
> will test
>>>> that.
>>>> 
>>>>  HBASE-2600 is a big change of encoding of meta, while my 
> understanding
>>>> is
>>>>  that 0.90->0,92 is a graceful HFile format conversion.  Are 
> there are
>>>>  things currently in trunk that further break compatiblity of the 
> file
>>>>  format? (what jiras?)
>>>> 
>>>>  Jon.
>>>> 
>>>> 
>>>>  On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl 
> <lhofhansl@yahoo.com>
>>>>  wrote:
>>>> 
>>>>  Not removing code for upgrade is fine.
>>>>  >
>>>>  >Todd, is Cloudera signing up for testing this path (0.90 to 
> 0.94)?
>>>>  >
>>>>  >
>>>>  >Stack, what's your feeling w.r.t. to HBASE-2600, will 
> keeping the 0.90
>>>>  -> 0.92 migration path
>>>>  >make the migration code for HBASE-2600 (much) more complicated 
> in
>>>> 0.94?
>>>>  >
>>>>  >
>>>>  >-- Lars
>>>>  >
>>>>  >
>>>>  >
>>>>  >----- Original Message -----
>>>>  >From: Stack <stack@duboce.net>
>>>>  >To: dev@hbase.apache.org
>>>>  >Cc:
>>>>  >Sent: Friday, January 27, 2012 9:02 AM
>>>>  >Subject: Re: hbase 0.94.0
>>>>  >
>>>>  >
>>>>  >On Fri, Jan 27, 2012 at 8:42 AM, Stack <stack@duboce.net> 
> wrote:
>>>>  >> In this particular case, there was no explicit migration 
> step needed
>>>>  >> going 0.90 to 0.92.  Upgrading from 0.90 to 0.94 should 
> just be
>>>>  >> running the 0.92 to 0.94 migration script (if there is 
> one).
>>>>  >>
>>>>  >
>>>>  >Thinking more, the above only really holds if we keep the 
> .META.
>>>>  >migration code that runs in 0.92 on startup on into 0.94 (I 
> have a
>>>>  >patch here where I'm removing it... I should put this bit 
> of code
>>>>  >back).
>>>>  >
>>>>  >I see Todd that you vote against removing hfile v1 in 0.94.  To 
> make
>>>>  >going from CDH3 to CDH4, we should not purge any migrating code 
> or old
>>>>  >version support.
>>>>  >
>>>>  >I'd be fine w/ that.  We'll need to work hard to ensure 
> it taking it
>>>>  >on as a principal for 0.94.  Ok w/ you "Iron Hand" 
> Lars?
>>>>  >
>>>>  >St.Ack
>>>>  >
>>>>  >
>>>> 
>>>> 
>>>>  --
>>>>  // Jonathan Hsieh (shay)
>>>>  // Software Engineer, Cloudera
>>>> 
>>>>  // jon@cloudera.com
>>>> 
>>> 
>> 
>> 
>> -- 
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
> 

Mime
View raw message