hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devaraj Das <d...@hortonworks.com>
Subject Re: hbase 0.94.0
Date Tue, 31 Jan 2012 07:53:52 GMT

On Jan 30, 2012, at 11:39 AM, Jonathan Hsieh wrote:

> Nicolas,
> 
> For point 2 and 3, there definitely is a group of us that have been
> informally discussing long term wire and format compatibility in
> conversations.  There are some documents that should be coming out soon to
> the lists from these chats. The initial goal of this some of is to define a
> vocabulary, to try to scope and phase-in changes necessary to make this
> happen (possibly parts in 0.94, 0.96, 0.98...), and to define possible
> goals and non-goals.
> 
> My hope is that a thought-out first cut will come from Jimmy and I in the
> next week or so so that we start having discussion to reach a consensus and
> then start implementing.
> 
> Some folks from another group (I'll let them announce themselves) should be
> producing a doc focused more on the rpc portion of this soon as well.
> 

Thanks for the hint, Jon (*smile*)

BTW I have opened - https://issues.apache.org/jira/browse/HBASE-5305 / HBASE-5306 to keep
track of the issues..

We will upload some docs/discussion-points there soon.


> Jon.
> 
> On Mon, Jan 30, 2012 at 11:21 AM, Nicolas Spiegelberg
> <nspiegelberg@fb.com>wrote:
> 
>> Currently, we at Facebook are developing mostly on 89 but also doing some
>> significant exploratory work on trunk.  I think that most of our
>> development will continue to be done on 89 in the near future.  We plan to
>> release some lower-risk projects on 94.  However, we won't even entertain
>> a wide-scale upgrade strategy until:
>> 
>> 1) The trunk has more master work.  Specifically 89-master related
>> features that we implemented for our grid architecture.  It sounds like
>> most of those features are rising in prominence in the Open Source
>> community as well (Ted from eBay & Todd from Cloudera, in particular).
>> Hopefully, we can collaborate on porting/implementation.
>> 2) Client/Server compatibility.  Even if we upgraded the servers to
>> 94/96/whatever, we still will have a lot of clients running 89 & we can't
>> simply stop/start all of them at the same time.  We need 89/trunk RPC
>> compat for client/server.
>> 3) Durable Server Upgrade.  It does bother me that people are very quick
>> to consider removing both RPC & persistent data compatibility from 90 to
>> trunk (a branch that we're still actively releasing minor upgrades for).
>> We will need the ability to mutate the HDFS & ZK persistent data from the
>> 89 format to trunk.  Specifically, work like HFileV1 reader is critical
>> for analysis if the upgrade script fails and we need to debug.
>> 
>> Instead of discussing what classes we can throw away, can we instead think
>> about a strategy for long-term, cross-version compatibility that minimally
>> hurts trunk development?  Is HFileV1's presence severely hindering another
>> feature?
>> 
>> On 1/27/12 6:11 PM, "Jonathan Hsieh" <jon@cloudera.com> wrote:
>> 
>>> Lars, Matt,
>>> 
>>> I like Matt's suggestion -- as long as it is not a burden let's keep it
>>> in.
>>> If it becomes one, let's do what is best for the project.
>>> 
>>> If Cloudera needs to do the upgrade path, we'll provide one.  If it
>>> doesn't
>>> go to the apache repo, we'll most likely open source it on github or
>>> something, or "keep it on the jira" like some of the repair ruby scripts.
>>> 
>>> Question -- to the Trend/SalesForce/FB folks can you talk about your plans
>>> for upgrades?  Are you guys moving from 0.89/0.90ish to 0.92 already?  or
>>> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
>>> as well?
>>> 
>>> Jon.
>>> 
>>> 
>>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mcorgan@hotpads.com>
>> wrote:
>>> 
>>>> The reason I brought it up is now that Mikhail committed the data block
>>>> encoding I was going to take a stab at adding the prefix trie encoding I
>>>> was working on this past summer.  My plan is to first make a minimally
>>>> invasive patch to prove correctness.  But, after that there will
>>>> probably
>>>> be some big performance gains to be had from reworking some of the
>>>> things
>>>> like KeyValueScanner which I would not have the courage to get working
>>>> with
>>>> v1.
>>>> 
>>>> So, that was why I asked, but all of that is still more hypothetical
>>>> than
>>>> real, and I don't even know if the first part will be done before
>>>> branching
>>>> .94 at the end of February.  Makes sense to me to not delete v1 until
>>>> there's a good reason to, which it doesn't look like we have yet.  If I
>>>> get
>>>> to the point where v1 is halting progress then we can reevaluate based
>>>> on
>>>> more specific issues.  Maybe none of the prefix trie will even make
>>>> .94...
>>>> 
>>>> ..sent from my phone
>>>> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lhofhansl@yahoo.com> wrote:
>>>> 
>>>>> Hey Jon,
>>>>> 
>>>>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90
to
>>>>> 0.94 upgrade path and then timing does not work out and nobody signs
>>>>> up for
>>>>> the testing because it 0.92 is more convenient we'd have gone through
>>>>> this
>>>>> for nothing.
>>>>> 
>>>>> So... Thinking about this more I am -1 on supporting an official
>>>>> upgrade
>>>>> path other then from one release to the next.
>>>>> That said, we do not have to break things intentionally.
>>>>> 
>>>>> 
>>>>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
>>>>> as we won't havethe same argument for 0.96 :)
>>>>> 
>>>>> And I am not aware of any file compatibility issues.
>>>>> 
>>>>> 
>>>>> We can also leave the 0.92 migration code in, but not officially
>>>>> support
>>>>> it as 0.90 to 0.94 path.
>>>>> Then CDH4 can make sure (if needed) that it is all working.
>>>>> 
>>>>> 
>>>>> Does that work for you guys?
>>>>> 
>>>>> -- Lars
>>>>> 
>>>>> 
>>>>> 
>>>>> ________________________________
>>>>> From: Jonathan Hsieh <jon@cloudera.com>
>>>>> To: dev@hbase.apache.org; lars hofhansl <lhofhansl@yahoo.com>
>>>>> Sent: Friday, January 27, 2012 10:10 AM
>>>>> Subject: Re: hbase 0.94.0
>>>>> 
>>>>> 
>>>>> Lars,
>>>>> 
>>>>> The upcoming CDH4 Beta HBase will be based on the latest hotness,
>>>>> 0.92.0.
>>>>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If
>>>>> that
>>>>> HBase is 0.92 based, we'll test that, and if timing works out and we
>>>>> decide
>>>>> 0.94, we'll have to have a path (0.90->0.94) for than and will test
>>>>> that.
>>>>> 
>>>>> HBASE-2600 is a big change of encoding of meta, while my understanding
>>>>> is
>>>>> that 0.90->0,92 is a graceful HFile format conversion.  Are there
are
>>>>> things currently in trunk that further break compatiblity of the file
>>>>> format? (what jiras?)
>>>>> 
>>>>> Jon.
>>>>> 
>>>>> 
>>>>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lhofhansl@yahoo.com>
>>>>> wrote:
>>>>> 
>>>>> Not removing code for upgrade is fine.
>>>>>> 
>>>>>> Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>>>>>> 
>>>>>> 
>>>>>> Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the
0.90
>>>>> -> 0.92 migration path
>>>>>> make the migration code for HBASE-2600 (much) more complicated in
>>>>> 0.94?
>>>>>> 
>>>>>> 
>>>>>> -- Lars
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ----- Original Message -----
>>>>>> From: Stack <stack@duboce.net>
>>>>>> To: dev@hbase.apache.org
>>>>>> Cc:
>>>>>> Sent: Friday, January 27, 2012 9:02 AM
>>>>>> Subject: Re: hbase 0.94.0
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jan 27, 2012 at 8:42 AM, Stack <stack@duboce.net> wrote:
>>>>>>> In this particular case, there was no explicit migration step
needed
>>>>>>> going 0.90 to 0.92.  Upgrading from 0.90 to 0.94 should just
be
>>>>>>> running the 0.92 to 0.94 migration script (if there is one).
>>>>>>> 
>>>>>> 
>>>>>> Thinking more, the above only really holds if we keep the .META.
>>>>>> migration code that runs in 0.92 on startup on into 0.94 (I have
a
>>>>>> patch here where I'm removing it... I should put this bit of code
>>>>>> back).
>>>>>> 
>>>>>> I see Todd that you vote against removing hfile v1 in 0.94.  To make
>>>>>> going from CDH3 to CDH4, we should not purge any migrating code or
old
>>>>>> version support.
>>>>>> 
>>>>>> I'd be fine w/ that.  We'll need to work hard to ensure it taking
it
>>>>>> on as a principal for 0.94.  Ok w/ you "Iron Hand" Lars?
>>>>>> 
>>>>>> St.Ack
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> // Jonathan Hsieh (shay)
>>>>> // Software Engineer, Cloudera
>>>>> 
>>>>> // jon@cloudera.com
>>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> // Jonathan Hsieh (shay)
>>> // Software Engineer, Cloudera
>>> // jon@cloudera.com
>> 
>> 
> 
> 
> -- 
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com


Mime
View raw message