Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: hbase 0.94.0
From: Devaraj Das <ddas@hortonworks.com>
In-Reply-To: 
 <CAAha9a28rTHpr4ZDQY_bkHGwiqqw895MncP936zkAADvVV9-4g@mail.gmail.com>
Date: Mon, 30 Jan 2012 23:53:52 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <C809263E-61A4-49CD-9C4D-3867A5C04DF6@hortonworks.com>
References: 
 <CAAha9a0OJKJpALyg78R1FVSQ7zqv42caqm9rQjMxNLiccqVrGQ@mail.gmail.com>
 <CB4C258B.2B26D%nspiegelberg@fb.com>
 <CAAha9a28rTHpr4ZDQY_bkHGwiqqw895MncP936zkAADvVV9-4g@mail.gmail.com>
To: dev@hbase.apache.org


On Jan 30, 2012, at 11:39 AM, Jonathan Hsieh wrote:

> Nicolas,
>=20
> For point 2 and 3, there definitely is a group of us that have been
> informally discussing long term wire and format compatibility in
> conversations.  There are some documents that should be coming out =
soon to
> the lists from these chats. The initial goal of this some of is to =
define a
> vocabulary, to try to scope and phase-in changes necessary to make =
this
> happen (possibly parts in 0.94, 0.96, 0.98...), and to define possible
> goals and non-goals.
>=20
> My hope is that a thought-out first cut will come from Jimmy and I in =
the
> next week or so so that we start having discussion to reach a =
consensus and
> then start implementing.
>=20
> Some folks from another group (I'll let them announce themselves) =
should be
> producing a doc focused more on the rpc portion of this soon as well.
>=20

Thanks for the hint, Jon (*smile*)

BTW I have opened - https://issues.apache.org/jira/browse/HBASE-5305 / =
HBASE-5306 to keep track of the issues..

We will upload some docs/discussion-points there soon.


> Jon.
>=20
> On Mon, Jan 30, 2012 at 11:21 AM, Nicolas Spiegelberg
> <nspiegelberg@fb.com>wrote:
>=20
>> Currently, we at Facebook are developing mostly on 89 but also doing =
some
>> significant exploratory work on trunk.  I think that most of our
>> development will continue to be done on 89 in the near future.  We =
plan to
>> release some lower-risk projects on 94.  However, we won't even =
entertain
>> a wide-scale upgrade strategy until:
>>=20
>> 1) The trunk has more master work.  Specifically 89-master related
>> features that we implemented for our grid architecture.  It sounds =
like
>> most of those features are rising in prominence in the Open Source
>> community as well (Ted from eBay & Todd from Cloudera, in =
particular).
>> Hopefully, we can collaborate on porting/implementation.
>> 2) Client/Server compatibility.  Even if we upgraded the servers to
>> 94/96/whatever, we still will have a lot of clients running 89 & we =
can't
>> simply stop/start all of them at the same time.  We need 89/trunk RPC
>> compat for client/server.
>> 3) Durable Server Upgrade.  It does bother me that people are very =
quick
>> to consider removing both RPC & persistent data compatibility from 90 =
to
>> trunk (a branch that we're still actively releasing minor upgrades =
for).
>> We will need the ability to mutate the HDFS & ZK persistent data from =
the
>> 89 format to trunk.  Specifically, work like HFileV1 reader is =
critical
>> for analysis if the upgrade script fails and we need to debug.
>>=20
>> Instead of discussing what classes we can throw away, can we instead =
think
>> about a strategy for long-term, cross-version compatibility that =
minimally
>> hurts trunk development?  Is HFileV1's presence severely hindering =
another
>> feature?
>>=20
>> On 1/27/12 6:11 PM, "Jonathan Hsieh" <jon@cloudera.com> wrote:
>>=20
>>> Lars, Matt,
>>>=20
>>> I like Matt's suggestion -- as long as it is not a burden let's keep =
it
>>> in.
>>> If it becomes one, let's do what is best for the project.
>>>=20
>>> If Cloudera needs to do the upgrade path, we'll provide one.  If it
>>> doesn't
>>> go to the apache repo, we'll most likely open source it on github or
>>> something, or "keep it on the jira" like some of the repair ruby =
scripts.
>>>=20
>>> Question -- to the Trend/SalesForce/FB folks can you talk about your =
plans
>>> for upgrades?  Are you guys moving from 0.89/0.90ish to 0.92 =
already?  or
>>> are you guys going to have to do the direct upgrade to 0.94 from =
0.90 land
>>> as well?
>>>=20
>>> Jon.
>>>=20
>>>=20
>>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mcorgan@hotpads.com>
>> wrote:
>>>=20
>>>> The reason I brought it up is now that Mikhail committed the data =
block
>>>> encoding I was going to take a stab at adding the prefix trie =
encoding I
>>>> was working on this past summer.  My plan is to first make a =
minimally
>>>> invasive patch to prove correctness.  But, after that there will
>>>> probably
>>>> be some big performance gains to be had from reworking some of the
>>>> things
>>>> like KeyValueScanner which I would not have the courage to get =
working
>>>> with
>>>> v1.
>>>>=20
>>>> So, that was why I asked, but all of that is still more =
hypothetical
>>>> than
>>>> real, and I don't even know if the first part will be done before
>>>> branching
>>>> .94 at the end of February.  Makes sense to me to not delete v1 =
until
>>>> there's a good reason to, which it doesn't look like we have yet.  =
If I
>>>> get
>>>> to the point where v1 is halting progress then we can reevaluate =
based
>>>> on
>>>> more specific issues.  Maybe none of the prefix trie will even make
>>>> .94...
>>>>=20
>>>> ..sent from my phone
>>>> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lhofhansl@yahoo.com> =
wrote:
>>>>=20
>>>>> Hey Jon,
>>>>>=20
>>>>> understood. Makes 0.94 hard, though. If we decide now to have a =
0.90 to
>>>>> 0.94 upgrade path and then timing does not work out and nobody =
signs
>>>>> up for
>>>>> the testing because it 0.92 is more convenient we'd have gone =
through
>>>>> this
>>>>> for nothing.
>>>>>=20
>>>>> So... Thinking about this more I am -1 on supporting an official
>>>>> upgrade
>>>>> path other then from one release to the next.
>>>>> That said, we do not have to break things intentionally.
>>>>>=20
>>>>>=20
>>>>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As =
long
>>>>> as we won't havethe same argument for 0.96 :)
>>>>>=20
>>>>> And I am not aware of any file compatibility issues.
>>>>>=20
>>>>>=20
>>>>> We can also leave the 0.92 migration code in, but not officially
>>>>> support
>>>>> it as 0.90 to 0.94 path.
>>>>> Then CDH4 can make sure (if needed) that it is all working.
>>>>>=20
>>>>>=20
>>>>> Does that work for you guys?
>>>>>=20
>>>>> -- Lars
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> ________________________________
>>>>> From: Jonathan Hsieh <jon@cloudera.com>
>>>>> To: dev@hbase.apache.org; lars hofhansl <lhofhansl@yahoo.com>
>>>>> Sent: Friday, January 27, 2012 10:10 AM
>>>>> Subject: Re: hbase 0.94.0
>>>>>=20
>>>>>=20
>>>>> Lars,
>>>>>=20
>>>>> The upcoming CDH4 Beta HBase will be based on the latest hotness,
>>>>> 0.92.0.
>>>>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. =
If
>>>>> that
>>>>> HBase is 0.92 based, we'll test that, and if timing works out and =
we
>>>>> decide
>>>>> 0.94, we'll have to have a path (0.90->0.94) for than and will =
test
>>>>> that.
>>>>>=20
>>>>> HBASE-2600 is a big change of encoding of meta, while my =
understanding
>>>>> is
>>>>> that 0.90->0,92 is a graceful HFile format conversion.  Are there =
are
>>>>> things currently in trunk that further break compatiblity of the =
file
>>>>> format? (what jiras?)
>>>>>=20
>>>>> Jon.
>>>>>=20
>>>>>=20
>>>>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl =
<lhofhansl@yahoo.com>
>>>>> wrote:
>>>>>=20
>>>>> Not removing code for upgrade is fine.
>>>>>>=20
>>>>>> Todd, is Cloudera signing up for testing this path (0.90 to =
0.94)?
>>>>>>=20
>>>>>>=20
>>>>>> Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the =
0.90
>>>>> -> 0.92 migration path
>>>>>> make the migration code for HBASE-2600 (much) more complicated in
>>>>> 0.94?
>>>>>>=20
>>>>>>=20
>>>>>> -- Lars
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> ----- Original Message -----
>>>>>> From: Stack <stack@duboce.net>
>>>>>> To: dev@hbase.apache.org
>>>>>> Cc:
>>>>>> Sent: Friday, January 27, 2012 9:02 AM
>>>>>> Subject: Re: hbase 0.94.0
>>>>>>=20
>>>>>>=20
>>>>>> On Fri, Jan 27, 2012 at 8:42 AM, Stack <stack@duboce.net> wrote:
>>>>>>> In this particular case, there was no explicit migration step =
needed
>>>>>>> going 0.90 to 0.92.  Upgrading from 0.90 to 0.94 should just be
>>>>>>> running the 0.92 to 0.94 migration script (if there is one).
>>>>>>>=20
>>>>>>=20
>>>>>> Thinking more, the above only really holds if we keep the .META.
>>>>>> migration code that runs in 0.92 on startup on into 0.94 (I have =
a
>>>>>> patch here where I'm removing it... I should put this bit of code
>>>>>> back).
>>>>>>=20
>>>>>> I see Todd that you vote against removing hfile v1 in 0.94.  To =
make
>>>>>> going from CDH3 to CDH4, we should not purge any migrating code =
or old
>>>>>> version support.
>>>>>>=20
>>>>>> I'd be fine w/ that.  We'll need to work hard to ensure it taking =
it
>>>>>> on as a principal for 0.94.  Ok w/ you "Iron Hand" Lars?
>>>>>>=20
>>>>>> St.Ack
>>>>>>=20
>>>>>>=20
>>>>>=20
>>>>>=20
>>>>> --
>>>>> // Jonathan Hsieh (shay)
>>>>> // Software Engineer, Cloudera
>>>>>=20
>>>>> // jon@cloudera.com
>>>>>=20
>>>>=20
>>>=20
>>>=20
>>> --
>>> // Jonathan Hsieh (shay)
>>> // Software Engineer, Cloudera
>>> // jon@cloudera.com
>>=20
>>=20
>=20
>=20
> --=20
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com