Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
References: 
 <CAAha9a0OJKJpALyg78R1FVSQ7zqv42caqm9rQjMxNLiccqVrGQ@mail.gmail.com>
 <CB4C258B.2B26D%nspiegelberg@fb.com>
Message-ID: <1327953685.93363.YahooMailNeo@web164506.mail.gq1.yahoo.com>
Date: Mon, 30 Jan 2012 12:01:25 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: Andrew Purtell <apurtell@apache.org>
Subject: Re: hbase 0.94.0
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
In-Reply-To: <CB4C258B.2B26D%nspiegelberg@fb.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

> From: Nicolas Spiegelberg <nspiegelberg@fb.com>=0A> 2) Client/Server comp=
atibility.=A0 Even if we upgraded the servers to=0A=0A> 94/96/whatever, we =
still will have a lot of clients running 89 & we =0A> can't=0A=0A=0A0.92+ s=
upports pluggable RPC engines. Can you manage 0.89 compat that way?=0A=0A=
=0A> 3) Durable Server Upgrade.=A0 It does bother me that people are very q=
uick=0A> to consider removing both RPC & persistent data compatibility from=
 90 to=0A> trunk (a branch that we're still actively releasing minor upgrad=
es for).=0A> We will need the ability to mutate the HDFS & ZK persistent da=
ta from the=0A> 89 format to trunk.=A0 Specifically, work like HFileV1 read=
er is critical=0A> for analysis if the upgrade script fails and we need to =
debug.=0A=0ATraditionally there have been no guarantees of cross major vers=
ion compatibility. RPC especially. Never a rolling upgrade from an 0.90 to =
an 0.92, for example. For persistent data, there is a migration utility for=
 upping from one major release to the next.=0A=0ARegarding RPC, this state =
of affairs is not really acceptable to anyone any more. Over in core there'=
s work to move 99% of RPC to protobufs, with only the thinnest Writable hea=
der. In this thread there seem several here who want to tackle this for HBa=
se now.=0A=0ARegarding major version data migrations, the attitude I believ=
e is pre 1.0 we can entertain design changes that break compatibility, in s=
earch of something that works well enough to be 1.0. From that point forwar=
d, compatibility is a requirement.=0A=0AIn practical terms, with long term =
scale deployments with compatibility requirements, I think we are quite nea=
r 1.0. Earlier I argued that we should adopt the label ("1.0") like Hadoop =
core did in concession to the realities of large scale production deploys, =
but there were good counter arguments against doing so, namely that HBase h=
as a few rough edges yet.=0A=0A> Instead of discussing what classes we can =
throw away, can we instead think=0A> about a strategy for long-term, cross-=
version compatibility that minimally=0A> hurts trunk development?=0A=0A+1=
=0A=0ABest regards,=0A=0A=A0 =A0 - Andy=0A=0AProblems worthy of attack prov=
e their worth by hitting back. - Piet Hein (via Tom White) =0A=0A=0A----- O=
riginal Message -----=0A> From: Nicolas Spiegelberg <nspiegelberg@fb.com>=
=0A> To: "dev@hbase.apache.org" <dev@hbase.apache.org>=0A> Cc: =0A> Sent: M=
onday, January 30, 2012 11:21 AM=0A> Subject: Re: hbase 0.94.0=0A> =0A> Cur=
rently, we at Facebook are developing mostly on 89 but also doing some=0A> =
significant exploratory work on trunk.=A0 I think that most of our=0A> deve=
lopment will continue to be done on 89 in the near future.=A0 We plan to=0A=
> release some lower-risk projects on 94.=A0 However, we won't even enterta=
in=0A> a wide-scale upgrade strategy until:=0A> =0A> 1) The trunk has more =
master work.=A0 Specifically 89-master related=0A> features that we impleme=
nted for our grid architecture.=A0 It sounds like=0A> most of those feature=
s are rising in prominence in the Open Source=0A> community as well (Ted fr=
om eBay & Todd from Cloudera, in particular).=0A> Hopefully, we can collabo=
rate on porting/implementation.=0A> 2) Client/Server compatibility.=A0 Even=
 if we upgraded the servers to=0A> 94/96/whatever, we still will have a lot=
 of clients running 89 & we =0A> can't=0A> simply stop/start all of them at=
 the same time.=A0 We need 89/trunk RPC=0A> compat for client/server.=0A> 3=
) Durable Server Upgrade.=A0 It does bother me that people are very quick=
=0A> to consider removing both RPC & persistent data compatibility from 90 =
to=0A> trunk (a branch that we're still actively releasing minor upgrades f=
or).=0A> We will need the ability to mutate the HDFS & ZK persistent data f=
rom the=0A> 89 format to trunk.=A0 Specifically, work like HFileV1 reader i=
s critical=0A> for analysis if the upgrade script fails and we need to debu=
g.=0A> =0A> Instead of discussing what classes we can throw away, can we in=
stead think=0A> about a strategy for long-term, cross-version compatibility=
 that minimally=0A> hurts trunk development?=A0 Is HFileV1's presence sever=
ely hindering another=0A> feature?=0A> =0A> On 1/27/12 6:11 PM, "Jonathan H=
sieh" <jon@cloudera.com> wrote:=0A> =0A>> Lars, Matt,=0A>> =0A>> I like Mat=
t's suggestion -- as long as it is not a burden let's keep =0A> it=0A>> in.=
=0A>>  If it becomes one, let's do what is best for the project.=0A>> =0A>>=
 If Cloudera needs to do the upgrade path, we'll provide one.=A0 If it=0A>>=
 doesn't=0A>> go to the apache repo, we'll most likely open source it on gi=
thub or=0A>> something, or "keep it on the jira" like some of the repair ru=
by =0A> scripts.=0A>> =0A>> Question -- to the Trend/SalesForce/FB folks ca=
n you talk about your plans=0A>> for upgrades?=A0 Are you guys moving from =
0.89/0.90ish to 0.92 already?=A0 or=0A>> are you guys going to have to do t=
he direct upgrade to 0.94 from 0.90 land=0A>> as well?=0A>> =0A>> Jon.=0A>>=
 =0A>> =0A>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mcorgan@hotpads=
.com> =0A> wrote:=0A>> =0A>>>  The reason I brought it up is now that Mikha=
il committed the data block=0A>>>  encoding I was going to take a stab at a=
dding the prefix trie encoding =0A> I=0A>>>  was working on this past summe=
r.=A0 My plan is to first make a minimally=0A>>>  invasive patch to prove c=
orrectness.=A0 But, after that there will=0A>>> probably=0A>>>  be some big=
 performance gains to be had from reworking some of the=0A>>> things=0A>>> =
 like KeyValueScanner which I would not have the courage to get working=0A>=
>> with=0A>>>  v1.=0A>>> =0A>>>  So, that was why I asked, but all of that =
is still more hypothetical=0A>>> than=0A>>>  real, and I don't even know if=
 the first part will be done before=0A>>> branching=0A>>>  .94 at the end o=
f February.=A0 Makes sense to me to not delete v1 until=0A>>>  there's a go=
od reason to, which it doesn't look like we have =0A> yet.=A0 If I=0A>>> ge=
t=0A>>>  to the point where v1 is halting progress then we can reevaluate b=
ased=0A>>> on=0A>>>  more specific issues.=A0 Maybe none of the prefix trie=
 will even make=0A>>> .94...=0A>>> =0A>>>  ..sent from my phone=0A>>>  On J=
an 27, 2012 1:55 PM, "lars hofhansl" =0A> <lhofhansl@yahoo.com> wrote:=0A>>=
> =0A>>>> =A0 Hey Jon,=0A>>>> =0A>>>>  understood. Makes 0.94 hard, though.=
 If we decide now to have a =0A> 0.90 to=0A>>>>  0.94 upgrade path and then=
 timing does not work out and nobody =0A> signs=0A>>>> up for=0A>>>>  the t=
esting because it 0.92 is more convenient we'd have gone =0A> through=0A>>>=
> this=0A>>>>  for nothing.=0A>>>> =0A>>>>  So... Thinking about this more =
I am -1 on supporting an official=0A>>>> upgrade=0A>>>>  path other then fr=
om one release to the next.=0A>>>>  That said, we do not have to break thin=
gs intentionally.=0A>>>> =0A>>>> =0A>>>>  I'm fine pushing HBASE-2600 and H=
File v1 removal out of 0.94... =0A> As long=0A>>>>  as we won't havethe sam=
e argument for 0.96 :)=0A>>>> =0A>>>>  And I am not aware of any file compa=
tibility issues.=0A>>>> =0A>>>> =0A>>>>  We can also leave the 0.92 migrati=
on code in, but not officially=0A>>>> support=0A>>>>  it as 0.90 to 0.94 pa=
th.=0A>>>>  Then CDH4 can make sure (if needed) that it is all working.=0A>=
>>> =0A>>>> =0A>>>>  Does that work for you guys?=0A>>>> =0A>>>>  -- Lars=
=0A>>>> =0A>>>> =0A>>>> =0A>>>>  ________________________________=0A>>>> =
=A0 From: Jonathan Hsieh <jon@cloudera.com>=0A>>>>  To: dev@hbase.apache.or=
g; lars hofhansl <lhofhansl@yahoo.com>=0A>>>>  Sent: Friday, January 27, 20=
12 10:10 AM=0A>>>>  Subject: Re: hbase 0.94.0=0A>>>> =0A>>>> =0A>>>>  Lars,=
=0A>>>> =0A>>>>  The upcoming CDH4 Beta HBase will be based on the latest h=
otness,=0A>>>> 0.92.0.=0A>>>>  A CDH4 GA HBase will have to have an upgrade=
 path from CDH3 HBase. =0A> If=0A>>>> that=0A>>>>  HBase is 0.92 based, we'=
ll test that, and if timing works out =0A> and we=0A>>>> decide=0A>>>>  0.9=
4, we'll have to have a path (0.90->0.94) for than and =0A> will test=0A>>>=
> that.=0A>>>> =0A>>>>  HBASE-2600 is a big change of encoding of meta, whi=
le my =0A> understanding=0A>>>> is=0A>>>>  that 0.90->0,92 is a graceful HF=
ile format conversion.=A0 Are =0A> there are=0A>>>>  things currently in tr=
unk that further break compatiblity of the =0A> file=0A>>>>  format? (what =
jiras?)=0A>>>> =0A>>>>  Jon.=0A>>>> =0A>>>> =0A>>>>  On Fri, Jan 27, 2012 a=
t 9:16 AM, lars hofhansl =0A> <lhofhansl@yahoo.com>=0A>>>>  wrote:=0A>>>> =
=0A>>>>  Not removing code for upgrade is fine.=0A>>>>  >=0A>>>>  >Todd, is=
 Cloudera signing up for testing this path (0.90 to =0A> 0.94)?=0A>>>>  >=
=0A>>>>  >=0A>>>>  >Stack, what's your feeling w.r.t. to HBASE-2600, will =
=0A> keeping the 0.90=0A>>>>  -> 0.92 migration path=0A>>>>  >make the migr=
ation code for HBASE-2600 (much) more complicated =0A> in=0A>>>> 0.94?=0A>>=
>>  >=0A>>>>  >=0A>>>>  >-- Lars=0A>>>>  >=0A>>>>  >=0A>>>>  >=0A>>>>  >---=
-- Original Message -----=0A>>>>  >From: Stack <stack@duboce.net>=0A>>>>  >=
To: dev@hbase.apache.org=0A>>>>  >Cc:=0A>>>>  >Sent: Friday, January 27, 20=
12 9:02 AM=0A>>>>  >Subject: Re: hbase 0.94.0=0A>>>>  >=0A>>>>  >=0A>>>>  >=
On Fri, Jan 27, 2012 at 8:42 AM, Stack <stack@duboce.net> =0A> wrote:=0A>>>=
>  >> In this particular case, there was no explicit migration =0A> step ne=
eded=0A>>>>  >> going 0.90 to 0.92.=A0 Upgrading from 0.90 to 0.94 should =
=0A> just be=0A>>>>  >> running the 0.92 to 0.94 migration script (if there=
 is =0A> one).=0A>>>>  >>=0A>>>>  >=0A>>>>  >Thinking more, the above only =
really holds if we keep the =0A> .META.=0A>>>>  >migration code that runs i=
n 0.92 on startup on into 0.94 (I =0A> have a=0A>>>>  >patch here where I'm=
 removing it... I should put this bit =0A> of code=0A>>>>  >back).=0A>>>>  =
>=0A>>>>  >I see Todd that you vote against removing hfile v1 in 0.94.=A0 T=
o =0A> make=0A>>>>  >going from CDH3 to CDH4, we should not purge any migra=
ting code =0A> or old=0A>>>>  >version support.=0A>>>>  >=0A>>>>  >I'd be f=
ine w/ that.=A0 We'll need to work hard to ensure =0A> it taking it=0A>>>> =
 >on as a principal for 0.94.=A0 Ok w/ you "Iron Hand" =0A> Lars?=0A>>>>  >=
=0A>>>>  >St.Ack=0A>>>>  >=0A>>>>  >=0A>>>> =0A>>>> =0A>>>>  --=0A>>>>  // =
Jonathan Hsieh (shay)=0A>>>>  // Software Engineer, Cloudera=0A>>>> =0A>>>>=
  // jon@cloudera.com=0A>>>> =0A>>> =0A>> =0A>> =0A>> -- =0A>> // Jonathan =
Hsieh (shay)=0A>> // Software Engineer, Cloudera=0A>> // jon@cloudera.com=
=0A>