hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <shv.had...@gmail.com>
Subject Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Date Tue, 13 Aug 2013 00:47:57 GMT
Ok. After installing protobuf 2.5.0 I can compile trunk.
But now I cannot compile Hadoop-2 branches. None of them.
So if I switch between branches I need to reinstall protobuf?

Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
versions?
I did not get definite impression there is.
If not it could be a pretty big disruption.

Thanks,
--Konst



On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>
> To build trunk now you need protoc 2.5.0 (the build will fail with a
> warning if you don't have it).
>
> We'd propagate this to the 2 branches once the precommit build is back to
> normal and see things are OK.
>
> Thanks.
>
>
> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> > protoc 2.5.0.
> >
> > thx
> >
> >
> > On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> >> I can take care of re-installing 2.4 and installing 2.5 in a different
> >> location. This would fix 2.0 branch builds as well.
> >> Thoughts?
> >>
> >> -Giri
> >>
> >>
> >> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >> >wrote:
> >>
> >> > Giri,
> >> >
> >> > first of all, thanks for installing protoc 2.5.0.
> >> >
> >> > I didn't know we were installing them as the only version and not
> >> driven by
> >> > env/path settings.
> >> >
> >> > Now we have a bit of a problem, precommit builds are broken because of
> >> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >> >
> >> > We have to options:
> >> >
> >> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
> any
> >> > follow up issues.
> >> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> >> 2.5.0
> >> > coexisting
> >> >
> >> > My take would be to commit HADOOP-9845 in trunk, iron out any issues
> an
> >> > then merge it to the other branches.
> >> >
> >> > We need to sort this out quickly as precommits are not working.
> >> >
> >> > I'll wait till 3PM today  for objections to option #1, if none I'll
> >> commit
> >> > it to trunk.
> >> >
> >> > Thanks.
> >> >
> >> > Alejandro
> >> >
> >> >
> >> >
> >> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> >> > gkesavan@hortonworks.com> wrote:
> >> >
> >> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
> default
> >> > path.
> >> > > If we still need 2.4 I may have to install it. Let me know
> >> > >
> >> > > -Giri
> >> > >
> >> > >
> >> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >> > > >wrote:
> >> > >
> >> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both
> so
> >> we
> >> > > can
> >> > > > use and env to set it in the jobs?
> >> > > >
> >> > > > thx
> >> > > >
> >> > > > Alejandro
> >> > > > (phone typing)
> >> > > >
> >> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> >> gkesavan@hortonworks.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > -Giri
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> >> > > > > gkesavan@hortonworks.com> wrote:
> >> > > > >
> >> > > > >> Alejandro,
> >> > > > >>
> >> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >> > > > >>
> >> > > > >> -Giri
> >> > > > >>
> >> > > > >>
> >> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> >> > tucu@cloudera.com
> >> > > > >wrote:
> >> > > > >>
> >> > > > >>> pinging again, I need help from somebody with sudo
access to
> the
> >> > > hadoop
> >> > > > >>> jenkins boxes to do this or to get sudo access for
a couple of
> >> > hours
> >> > > to
> >> > > > >>> set
> >> > > > >>> up myself.
> >> > > > >>>
> >> > > > >>> Please!!!
> >> > > > >>>
> >> > > > >>> thx
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur
<
> >> > > tucu@cloudera.com
> >> > > > >>>> wrote:
> >> > > > >>>
> >> > > > >>>> To move forward with this we need protoc 2.5.0
in the apache
> >> > hadoop
> >> > > > >>>> jenkins boxes.
> >> > > > >>>>
> >> > > > >>>> Who can help with this? I assume somebody at
Y!, right?
> >> > > > >>>>
> >> > > > >>>> Thx
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark
<
> >> eclark@apache.org>
> >> > > > >>> wrote:
> >> > > > >>>>
> >> > > > >>>>> In HBase land we've pretty well discovered
that we'll need
> to
> >> > have
> >> > > > the
> >> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR
servers are
> >> > running.
> >> > > > >>>>> That is to say there are issues with ever
having 2.4.x and
> >> 2.5.x
> >> > on
> >> > > > >>>>> the same class path.
> >> > > > >>>>>
> >> > > > >>>>> Upgrading to 2.5.x would be great, as it
brings some new
> >> classes
> >> > we
> >> > > > >>>>> could use.  With that said HBase is getting
pretty close to
> a
> >> > > rather
> >> > > > >>>>> large release (0.96.0 aka The Singularity)
so getting this
> in
> >> > > sooner
> >> > > > >>>>> rather than later would be great.  If we
could get this into
> >> > 2.1.0
> >> > > it
> >> > > > >>>>> would be great as that would allow us to
have a pretty easy
> >> story
> >> > > to
> >> > > > >>>>> users with regards to protobuf version.
> >> > > > >>>>>
> >> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee
<
> >> kihwal@yahoo-inc.com
> >> > >
> >> > > > >>> wrote:
> >> > > > >>>>>> Sorry to hijack the thread but, I also
wanted to mention
> >> Avro.
> >> > See
> >> > > > >>>>> HADOOP-9672.
> >> > > > >>>>>> The version we are using has memory
leak and inefficiency
> >> > issues.
> >> > > > >>> We've
> >> > > > >>>>> seen users running into it.
> >> > > > >>>>>>
> >> > > > >>>>>> Kihwal
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> ________________________________
> >> > > > >>>>>> From: Tsuyoshi OZAWA <ozawa.tsuyoshi@gmail.com>
> >> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> >> > common-dev@hadoop.apache.org>
> >> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
> hdfs-dev@hadoop.apache.org
> >> >;
> >> > "
> >> > > > >>>>> yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>;
"
> >> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> >> > mapreduce-dev@hadoop.apache.org>
> >> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59
AM
> >> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0
for the 2.1.0
> release,
> >> > > > >>>>> HADOOP-9845
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> Hi,
> >> > > > >>>>>>
> >> > > > >>>>>> About Hadoop, Harsh is dealing with
this problem in
> >> HADOOP-9346.
> >> > > > >>>>>> For more detail, please see the JIRA
ticket:
> >> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >> > > > >>>>>>
> >> > > > >>>>>> - Tsuyoshi
> >> > > > >>>>>>
> >> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro
Abdelnur <
> >> > > > >>> tucu@cloudera.com>
> >> > > > >>>>> wrote:
> >> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0
for the 2.1.0
> release.
> >> > > > >>>>>>>
> >> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf
2.5 has significant
> >> > > benefits
> >> > > > >>> to
> >> > > > >>>>>>> justify the upgrade.
> >> > > > >>>>>>>
> >> > > > >>>>>>> Doing the upgrade now, with the
first beta, will make
> things
> >> > > easier
> >> > > > >>> for
> >> > > > >>>>>>> downstream projects (like HBase)
using protobuf and
> adopting
> >> > > Hadoop
> >> > > > >>> 2.
> >> > > > >>>>> If
> >> > > > >>>>>>> we do the upgrade later, downstream
projects will have to
> >> > > support 2
> >> > > > >>>>>>> different versions and they my get
in nasty waters due to
> >> > > classpath
> >> > > > >>>>> issues.
> >> > > > >>>>>>>
> >> > > > >>>>>>> I've locally tested the patch in
a pseudo deployment of
> >> > > 2.1.0-beta
> >> > > > >>>>> branch
> >> > > > >>>>>>> and it works fine (something is
broken in trunk in the RPC
> >> > layer
> >> > > > >>>>> YARN-885).
> >> > > > >>>>>>>
> >> > > > >>>>>>> Now, to do this it will require
a few things:
> >> > > > >>>>>>>
> >> > > > >>>>>>> * Make sure protobuf 2.5.0 is available
in the jenkins box
> >> > > > >>>>>>> * A follow up email to dev@ aliases
indicating developers
> >> > should
> >> > > > >>>>> install
> >> > > > >>>>>>> locally protobuf 2.5.0
> >> > > > >>>>>>>
> >> > > > >>>>>>> Thanks.
> >> > > > >>>>>>>
> >> > > > >>>>>>> --
> >> > > > >>>>>>> Alejandro
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> --
> >> > > > >>>> Alejandro
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> --
> >> > > > >>> Alejandro
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Alejandro
> >> >
> >>
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Alejandro
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message