hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject [UPDATE 3] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Date Wed, 14 Aug 2013 22:08:59 GMT
test-patch came back.

I'll commit to trunk and all "2" branches.

Once done I'll send an email indicating new protoc is required for
development.

Thanks.


On Wed, Aug 14, 2013 at 10:51 AM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> I've filed https://issues.apache.org/jira/browse/HADOOP-9872 addressing
> the following:
>
> ---------
>
>    - handles protoc version correctly independently of the exit code
>    - if HADOOP_PROTOC_PATH env var is defined, it uses it as the protoc
>    executable * if HADOOP_PROTOC_PATH is not defined, it picks protoc from the
>    PATH
>    - documentation updated to reflect 2.5.0 is required
>    - enforces the version of protoc and protobuf JAR are the same
>    - Added to VersionInfo the protoc version used (sooner or later this
>    will be useful for in a troubleshooting situation).
>
> Luke Lu<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vicaya> suggested
> to make the version check for protoc lax (i.e. 2.5.*). While working on the
> patch I've thought about that. But that would introduce a potential
> mismatch between protoc and protobuff JAR.
>
> Still If you want to use different version of protoc/protobuff from the
> one defined in the POM, you can use the -Dprotobuf.version=#### to specify
> your alternate version. But I would recommend not to do this, because if
> you publish the artifacts to a Maven repo, the fact you used
> -Dprotobuf.version=#### will be lost and the version defined in the POM
> properties will be used (IMO Maven should use the effective POM on deploy,
> but they don't).
>
> ---------
>
> It would be great if a few people test the patch locally.
>
> Once this is committed to trunk I'll bacport HADOOP-9845 & HADOOP-9872 to
> all the 2 branches.
>
> Thx.
>
> On Tue, Aug 13, 2013 at 1:09 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:
>
>>
>> There is no indication that protoc 2.5.0 is breaking anything.
>>
>> Hadoop-trunk builds have been failing way before 1/2 way with:
>>
>> -------
>>
>>
>> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException; nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash
or System.exit called ? -> [Help 1]
>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException; nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash
or System.exit called ?
>>
>> -------
>>
>>
>> The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards
>> the end of mapreduce tests.
>>
>> Until then there were no failures at all.
>>
>> I've increased heap size and tried a second run and the failure was
>> earlier.
>>
>> I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has
>> been failing the same way in all the kept builds.
>>
>> We need to fix Hadoop-trunk builds independently of this.
>>
>> Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta
>> branches to get all the other jenkins jobs working?
>>
>> I'll wait till tomorrow morning before proceeding.
>>
>> Thx
>>
>>
>>
>>
>>  On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:
>>
>>> Jenkins is running a full test run on trunk using protoc 2.5.0.
>>>
>>>   https://builds.apache.org/job/Hadoop-trunk/480
>>>
>>> And it seems go be going just fine.
>>>
>>> If everything looks OK, I'm planing to backport HADOOP-9845 to the
>>> 2.1.0-beta branch midday PST tomorrow. This will normalize all builds
>>> failures do the protoc mismatch.
>>>
>>> Thanks.
>>>
>>> Alejandro
>>>
>>>
>>> On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:
>>>
>>>> shooting to get it i n for 2.1.0.
>>>>
>>>> at moment is in trunk till the nightly finishes. then we'll decide
>>>>
>>>> in the mean time, you can have multiple versions installed in diff dirs
>>>> and set the right one in the path
>>>>
>>>> thx
>>>>
>>>> Alejandro
>>>> (phone typing)
>>>>
>>>> On Aug 12, 2013, at 17:47, Konstantin Shvachko <shv.hadoop@gmail.com>
>>>> wrote:
>>>>
>>>> > Ok. After installing protobuf 2.5.0 I can compile trunk.
>>>> > But now I cannot compile Hadoop-2 branches. None of them.
>>>> > So if I switch between branches I need to reinstall protobuf?
>>>> >
>>>> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
>>>> > versions?
>>>> > I did not get definite impression there is.
>>>> > If not it could be a pretty big disruption.
>>>> >
>>>> > Thanks,
>>>> > --Konst
>>>> >
>>>> >
>>>> >
>>>> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com>wrote:
>>>> >
>>>> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>>>> >>
>>>> >> To build trunk now you need protoc 2.5.0 (the build will fail with
a
>>>> >> warning if you don't have it).
>>>> >>
>>>> >> We'd propagate this to the 2 branches once the precommit build is
>>>> back to
>>>> >> normal and see things are OK.
>>>> >>
>>>> >> Thanks.
>>>> >>
>>>> >>
>>>> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>> >>> wrote:
>>>> >>
>>>> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make
>>>> trunk use
>>>> >>> protoc 2.5.0.
>>>> >>>
>>>> >>> thx
>>>> >>>
>>>> >>>
>>>> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>>>> >>> gkesavan@hortonworks.com> wrote:
>>>> >>>
>>>> >>>> I can take care of re-installing 2.4 and installing 2.5
in a
>>>> different
>>>> >>>> location. This would fix 2.0 branch builds as well.
>>>> >>>> Thoughts?
>>>> >>>>
>>>> >>>> -Giri
>>>> >>>>
>>>> >>>>
>>>> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>> >>>>> wrote:
>>>> >>>>
>>>> >>>>> Giri,
>>>> >>>>>
>>>> >>>>> first of all, thanks for installing protoc 2.5.0.
>>>> >>>>>
>>>> >>>>> I didn't know we were installing them as the only version
and not
>>>> >>>> driven by
>>>> >>>>> env/path settings.
>>>> >>>>>
>>>> >>>>> Now we have a bit of a problem, precommit builds are
broken
>>>> because of
>>>> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>>>> >>>>>
>>>> >>>>> We have to options:
>>>> >>>>>
>>>> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0
and iron
>>>> out
>>>> >> any
>>>> >>>>> follow up issues.
>>>> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and
have 2.4.1
>>>> and
>>>> >>>> 2.5.0
>>>> >>>>> coexisting
>>>> >>>>>
>>>> >>>>> My take would be to commit HADOOP-9845 in trunk, iron
out any
>>>> issues
>>>> >> an
>>>> >>>>> then merge it to the other branches.
>>>> >>>>>
>>>> >>>>> We need to sort this out quickly as precommits are not
working.
>>>> >>>>>
>>>> >>>>> I'll wait till 3PM today  for objections to option #1,
if none
>>>> I'll
>>>> >>>> commit
>>>> >>>>> it to trunk.
>>>> >>>>>
>>>> >>>>> Thanks.
>>>> >>>>>
>>>> >>>>> Alejandro
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan
<
>>>> >>>>> gkesavan@hortonworks.com> wrote:
>>>> >>>>>
>>>> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5.
2.5 is in the
>>>> >> default
>>>> >>>>> path.
>>>> >>>>>> If we still need 2.4 I may have to install it. Let
me know
>>>> >>>>>>
>>>> >>>>>> -Giri
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur
<
>>>> >>>> tucu@cloudera.com
>>>> >>>>>>> wrote:
>>>> >>>>>>
>>>> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what
is the path to both
>>>> >> so
>>>> >>>> we
>>>> >>>>>> can
>>>> >>>>>>> use and env to set it in the jobs?
>>>> >>>>>>>
>>>> >>>>>>> thx
>>>> >>>>>>>
>>>> >>>>>>> Alejandro
>>>> >>>>>>> (phone typing)
>>>> >>>>>>>
>>>> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan
<
>>>> >>>> gkesavan@hortonworks.com
>>>> >>>>>>
>>>> >>>>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc
2.5.0
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> -Giri
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan
Kesavan <
>>>> >>>>>>>> gkesavan@hortonworks.com> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>> Alejandro,
>>>> >>>>>>>>>
>>>> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>>> >>>>>>>>>
>>>> >>>>>>>>> -Giri
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro
Abdelnur <
>>>> >>>>> tucu@cloudera.com
>>>> >>>>>>>> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>>> pinging again, I need help from
somebody with sudo access to
>>>> >> the
>>>> >>>>>> hadoop
>>>> >>>>>>>>>> jenkins boxes to do this or to get
sudo access for a couple
>>>> of
>>>> >>>>> hours
>>>> >>>>>> to
>>>> >>>>>>>>>> set
>>>> >>>>>>>>>> up myself.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Please!!!
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> thx
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM,
Alejandro Abdelnur <
>>>> >>>>>> tucu@cloudera.com
>>>> >>>>>>>>>>> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>> To move forward with this we
need protoc 2.5.0 in the apache
>>>> >>>>> hadoop
>>>> >>>>>>>>>>> jenkins boxes.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Who can help with this? I assume
somebody at Y!, right?
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Thx
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24
PM, Elliott Clark <
>>>> >>>> eclark@apache.org>
>>>> >>>>>>>>>> wrote:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>> In HBase land we've pretty
well discovered that we'll need
>>>> >> to
>>>> >>>>> have
>>>> >>>>>>> the
>>>> >>>>>>>>>>>> same version of protobuf
that the HDFS/Yarn/MR servers are
>>>> >>>>> running.
>>>> >>>>>>>>>>>> That is to say there are
issues with ever having 2.4.x and
>>>> >>>> 2.5.x
>>>> >>>>> on
>>>> >>>>>>>>>>>> the same class path.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Upgrading to 2.5.x would
be great, as it brings some new
>>>> >>>> classes
>>>> >>>>> we
>>>> >>>>>>>>>>>> could use.  With that said
HBase is getting pretty close to
>>>> >> a
>>>> >>>>>> rather
>>>> >>>>>>>>>>>> large release (0.96.0 aka
The Singularity) so getting this
>>>> >> in
>>>> >>>>>> sooner
>>>> >>>>>>>>>>>> rather than later would
be great.  If we could get this
>>>> into
>>>> >>>>> 2.1.0
>>>> >>>>>> it
>>>> >>>>>>>>>>>> would be great as that would
allow us to have a pretty easy
>>>> >>>> story
>>>> >>>>>> to
>>>> >>>>>>>>>>>> users with regards to protobuf
version.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18
AM, Kihwal Lee <
>>>> >>>> kihwal@yahoo-inc.com
>>>> >>>>>>
>>>> >>>>>>>>>> wrote:
>>>> >>>>>>>>>>>>> Sorry to hijack the
thread but, I also wanted to mention
>>>> >>>> Avro.
>>>> >>>>> See
>>>> >>>>>>>>>>>> HADOOP-9672.
>>>> >>>>>>>>>>>>> The version we are using
has memory leak and inefficiency
>>>> >>>>> issues.
>>>> >>>>>>>>>> We've
>>>> >>>>>>>>>>>> seen users running into
it.
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Kihwal
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> ________________________________
>>>> >>>>>>>>>>>>> From: Tsuyoshi OZAWA
<ozawa.tsuyoshi@gmail.com>
>>>> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org"
<
>>>> >>>>> common-dev@hadoop.apache.org>
>>>> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org"
<
>>>> >> hdfs-dev@hadoop.apache.org
>>>> >>>>> ;
>>>> >>>>> "
>>>> >>>>>>>>>>>> yarn-dev@hadoop.apache.org"
<yarn-dev@hadoop.apache.org>;
>>>> "
>>>> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org"
<
>>>> >>>>> mapreduce-dev@hadoop.apache.org>
>>>> >>>>>>>>>>>>> Sent: Thursday, August
8, 2013 1:59 AM
>>>> >>>>>>>>>>>>> Subject: Re: Upgrade
to protobuf 2.5.0 for the 2.1.0
>>>> >> release,
>>>> >>>>>>>>>>>> HADOOP-9845
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Hi,
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> About Hadoop, Harsh
is dealing with this problem in
>>>> >>>> HADOOP-9346.
>>>> >>>>>>>>>>>>> For more detail, please
see the JIRA ticket:
>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> - Tsuyoshi
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> On Thu, Aug 8, 2013
at 1:49 AM, Alejandro Abdelnur <
>>>> >>>>>>>>>> tucu@cloudera.com>
>>>> >>>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>>>> I' like to upgrade
to protobuf 2.5.0 for the 2.1.0
>>>> >> release.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> As mentioned in
HADOOP-9845, Protobuf 2.5 has significant
>>>> >>>>>> benefits
>>>> >>>>>>>>>> to
>>>> >>>>>>>>>>>>>> justify the upgrade.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Doing the upgrade
now, with the first beta, will make
>>>> >> things
>>>> >>>>>> easier
>>>> >>>>>>>>>> for
>>>> >>>>>>>>>>>>>> downstream projects
(like HBase) using protobuf and
>>>> >> adopting
>>>> >>>>>> Hadoop
>>>> >>>>>>>>>> 2.
>>>> >>>>>>>>>>>> If
>>>> >>>>>>>>>>>>>> we do the upgrade
later, downstream projects will have to
>>>> >>>>>> support 2
>>>> >>>>>>>>>>>>>> different versions
and they my get in nasty waters due to
>>>> >>>>>> classpath
>>>> >>>>>>>>>>>> issues.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> I've locally tested
the patch in a pseudo deployment of
>>>> >>>>>> 2.1.0-beta
>>>> >>>>>>>>>>>> branch
>>>> >>>>>>>>>>>>>> and it works fine
(something is broken in trunk in the
>>>> RPC
>>>> >>>>> layer
>>>> >>>>>>>>>>>> YARN-885).
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Now, to do this
it will require a few things:
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> * Make sure protobuf
2.5.0 is available in the jenkins
>>>> box
>>>> >>>>>>>>>>>>>> * A follow up email
to dev@ aliases indicating
>>>> developers
>>>> >>>>> should
>>>> >>>>>>>>>>>> install
>>>> >>>>>>>>>>>>>> locally protobuf
2.5.0
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Thanks.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> --
>>>> >>>>>>>>>>>>>> Alejandro
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> --
>>>> >>>>>>>>>>> Alejandro
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> --
>>>> >>>>>>>>>> Alejandro
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> Alejandro
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Alejandro
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Alejandro
>>>> >>
>>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>>
>> --
>> Alejandro
>>
>
>
>
> --
> Alejandro
>



-- 
Alejandro

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message