hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Date Tue, 13 Aug 2013 03:35:07 GMT
Jenkins is running a full test run on trunk using protoc 2.5.0.

  https://builds.apache.org/job/Hadoop-trunk/480

And it seems go be going just fine.

If everything looks OK, I'm planing to backport HADOOP-9845 to the
2.1.0-beta branch midday PST tomorrow. This will normalize all builds
failures do the protoc mismatch.

Thanks.

Alejandro

On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> shooting to get it i n for 2.1.0.
>
> at moment is in trunk till the nightly finishes. then we'll decide
>
> in the mean time, you can have multiple versions installed in diff dirs
> and set the right one in the path
>
> thx
>
> Alejandro
> (phone typing)
>
> On Aug 12, 2013, at 17:47, Konstantin Shvachko <shv.hadoop@gmail.com>
> wrote:
>
> > Ok. After installing protobuf 2.5.0 I can compile trunk.
> > But now I cannot compile Hadoop-2 branches. None of them.
> > So if I switch between branches I need to reinstall protobuf?
> >
> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
> > versions?
> > I did not get definite impression there is.
> > If not it could be a pretty big disruption.
> >
> > Thanks,
> > --Konst
> >
> >
> >
> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
> >>
> >> To build trunk now you need protoc 2.5.0 (the build will fail with a
> >> warning if you don't have it).
> >>
> >> We'd propagate this to the 2 branches once the precommit build is back
> to
> >> normal and see things are OK.
> >>
> >> Thanks.
> >>
> >>
> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>> wrote:
> >>
> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk
> use
> >>> protoc 2.5.0.
> >>>
> >>> thx
> >>>
> >>>
> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> >>> gkesavan@hortonworks.com> wrote:
> >>>
> >>>> I can take care of re-installing 2.4 and installing 2.5 in a different
> >>>> location. This would fix 2.0 branch builds as well.
> >>>> Thoughts?
> >>>>
> >>>> -Giri
> >>>>
> >>>>
> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
> tucu@cloudera.com
> >>>>> wrote:
> >>>>
> >>>>> Giri,
> >>>>>
> >>>>> first of all, thanks for installing protoc 2.5.0.
> >>>>>
> >>>>> I didn't know we were installing them as the only version and not
> >>>> driven by
> >>>>> env/path settings.
> >>>>>
> >>>>> Now we have a bit of a problem, precommit builds are broken because
> of
> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >>>>>
> >>>>> We have to options:
> >>>>>
> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron
out
> >> any
> >>>>> follow up issues.
> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1
and
> >>>> 2.5.0
> >>>>> coexisting
> >>>>>
> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
> >> an
> >>>>> then merge it to the other branches.
> >>>>>
> >>>>> We need to sort this out quickly as precommits are not working.
> >>>>>
> >>>>> I'll wait till 3PM today  for objections to option #1, if none I'll
> >>>> commit
> >>>>> it to trunk.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> Alejandro
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> >>>>> gkesavan@hortonworks.com> wrote:
> >>>>>
> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
> >> default
> >>>>> path.
> >>>>>> If we still need 2.4 I may have to install it. Let me know
> >>>>>>
> >>>>>> -Giri
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path
to both
> >> so
> >>>> we
> >>>>>> can
> >>>>>>> use and env to set it in the jobs?
> >>>>>>>
> >>>>>>> thx
> >>>>>>>
> >>>>>>> Alejandro
> >>>>>>> (phone typing)
> >>>>>>>
> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> >>>> gkesavan@hortonworks.com
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Giri
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan
<
> >>>>>>>> gkesavan@hortonworks.com> wrote:
> >>>>>>>>
> >>>>>>>>> Alejandro,
> >>>>>>>>>
> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >>>>>>>>>
> >>>>>>>>> -Giri
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur
<
> >>>>> tucu@cloudera.com
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> pinging again, I need help from somebody with
sudo access to
> >> the
> >>>>>> hadoop
> >>>>>>>>>> jenkins boxes to do this or to get sudo access
for a couple of
> >>>>> hours
> >>>>>> to
> >>>>>>>>>> set
> >>>>>>>>>> up myself.
> >>>>>>>>>>
> >>>>>>>>>> Please!!!
> >>>>>>>>>>
> >>>>>>>>>> thx
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur
<
> >>>>>> tucu@cloudera.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> To move forward with this we need protoc
2.5.0 in the apache
> >>>>> hadoop
> >>>>>>>>>>> jenkins boxes.
> >>>>>>>>>>>
> >>>>>>>>>>> Who can help with this? I assume somebody
at Y!, right?
> >>>>>>>>>>>
> >>>>>>>>>>> Thx
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott
Clark <
> >>>> eclark@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> In HBase land we've pretty well discovered
that we'll need
> >> to
> >>>>> have
> >>>>>>> the
> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR
servers are
> >>>>> running.
> >>>>>>>>>>>> That is to say there are issues with
ever having 2.4.x and
> >>>> 2.5.x
> >>>>> on
> >>>>>>>>>>>> the same class path.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as
it brings some new
> >>>> classes
> >>>>> we
> >>>>>>>>>>>> could use.  With that said HBase is
getting pretty close to
> >> a
> >>>>>> rather
> >>>>>>>>>>>> large release (0.96.0 aka The Singularity)
so getting this
> >> in
> >>>>>> sooner
> >>>>>>>>>>>> rather than later would be great.  If
we could get this into
> >>>>> 2.1.0
> >>>>>> it
> >>>>>>>>>>>> would be great as that would allow us
to have a pretty easy
> >>>> story
> >>>>>> to
> >>>>>>>>>>>> users with regards to protobuf version.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal
Lee <
> >>>> kihwal@yahoo-inc.com
> >>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>> Sorry to hijack the thread but,
I also wanted to mention
> >>>> Avro.
> >>>>> See
> >>>>>>>>>>>> HADOOP-9672.
> >>>>>>>>>>>>> The version we are using has memory
leak and inefficiency
> >>>>> issues.
> >>>>>>>>>> We've
> >>>>>>>>>>>> seen users running into it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Kihwal
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ________________________________
> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <ozawa.tsuyoshi@gmail.com>
> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org"
<
> >>>>> common-dev@hadoop.apache.org>
> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org"
<
> >> hdfs-dev@hadoop.apache.org
> >>>>> ;
> >>>>> "
> >>>>>>>>>>>> yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>;
"
> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
> >>>>> mapreduce-dev@hadoop.apache.org>
> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59
AM
> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf
2.5.0 for the 2.1.0
> >> release,
> >>>>>>>>>>>> HADOOP-9845
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with
this problem in
> >>>> HADOOP-9346.
> >>>>>>>>>>>>> For more detail, please see the
JIRA ticket:
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Tsuyoshi
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM,
Alejandro Abdelnur <
> >>>>>>>>>> tucu@cloudera.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> I' like to upgrade to protobuf
2.5.0 for the 2.1.0
> >> release.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As mentioned in HADOOP-9845,
Protobuf 2.5 has significant
> >>>>>> benefits
> >>>>>>>>>> to
> >>>>>>>>>>>>>> justify the upgrade.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Doing the upgrade now, with
the first beta, will make
> >> things
> >>>>>> easier
> >>>>>>>>>> for
> >>>>>>>>>>>>>> downstream projects (like HBase)
using protobuf and
> >> adopting
> >>>>>> Hadoop
> >>>>>>>>>> 2.
> >>>>>>>>>>>> If
> >>>>>>>>>>>>>> we do the upgrade later, downstream
projects will have to
> >>>>>> support 2
> >>>>>>>>>>>>>> different versions and they
my get in nasty waters due to
> >>>>>> classpath
> >>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've locally tested the patch
in a pseudo deployment of
> >>>>>> 2.1.0-beta
> >>>>>>>>>>>> branch
> >>>>>>>>>>>>>> and it works fine (something
is broken in trunk in the RPC
> >>>>> layer
> >>>>>>>>>>>> YARN-885).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Now, to do this it will require
a few things:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is
available in the jenkins box
> >>>>>>>>>>>>>> * A follow up email to dev@
aliases indicating developers
> >>>>> should
> >>>>>>>>>>>> install
> >>>>>>>>>>>>>> locally protobuf 2.5.0
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Alejandro
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Alejandro
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Alejandro
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Alejandro
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
> >>
> >> --
> >> Alejandro
> >>
>



-- 
Alejandro

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message