giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francesco Sclano <>
Subject Re: I'm stucked with this error: Error starting MRAppMaster java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
Date Mon, 06 Aug 2018 09:55:15 GMT
Hi Eli,
first of all thank you for your support and your time!
I tryed to use giraph-dist-1.2.0-hadoop2-bin.tar.gz with hadoop-2.3.0 but I
got always the same error. I read all previous posts but I didn't find any
solution. I found this post about my error but I didn't find any occurrence of
"google protocol buffer jar with version 2.4.1" in my environment. Maybe
giraph-dist-1.2.0-bin.tar.gz was wrongly build with 2.4.1 dependency in
place of 2.5.0?! I tryed to download and
build it but the build fails on giraph-core. I used "mvn -Phadoop_2
-DskipTests clean install". Could you please provide me the right mvn
command or the right mvn configuration to be used?

*Hence I tryed to use giraph-dist-1.2.0-bin.tar.gz with hadoop-1.2.1 and
this time it properly works: I succeed to use giraph in a ec2 cluster with
this input graph
<> (for beginning 1
namenode and 2 datenodes with free micro tier - ubuntu 16.04). I didn't
change anything in my giraph parameters!!!* Anyway I'd prefer to use
giraph-1.2 with hadoop-2.

I found this
and also the documentation below in, but
I'm still a bit confused. In order to build giraph1.2 with hadoop-2.5.1 I
have to use "mvn -Phadoop_2 clean package -DskipTests" or I have to use
"mvn -Phadoop_yarn -Dhadoop.version=2.5.1 clean package -DskipTests". Why
both builds fail on giraph-core? Is giraph-1.2 compatible with hadoop-2.5.1
without using yarn? How hadoop-2.5.1 can be configured without using yarn?

- Apache Hadoop 2 (latest version: 2.5.1)

  This is the latest version of Hadoop 2 (supporting YARN in addition
  to MapReduce) Giraph could use. You may tell maven to use this version
  with "mvn -Phadoop_2 <goals>".

2018-08-01 12:11 GMT+02:00 Eli Reisman <>:

> hi francesco,
> I think in the mailing list you might find some tips for running on hadoop
> 2.5.x but IIRC hadoop 2.3.x plus appropriate Giraph build works. Again,
> given the error msg I'd also carefully review your command line options and
> config file just in case.
> Also for context as far as I know no one is actively maintaining Giraph on
> YARN compatibility for newer versions right now so taking a look at the
> mailing list backscroll will probably yield the best info on which
> combinations work best
> On Mon, Jul 30, 2018, 11:52 AM Francesco Sclano <
>> wrote:
>> Hi Eli,
>> ok, thanks for you suggestion. I'll try it and let you know. In order to
>> be sure, before try to use an earlier version of yarn/hadoop2, I'm using
>> giraph-dist-1.2.0-hadoop2-bin.tar.gz. Does this giraph version
>> officially support hadoop-2.5.1 with yarn? If not how I have to configure
>> hadoop-2.5.1? Thanks
>> 2018-07-29 9:47 GMT+02:00 Eli Reisman <>:
>>> hey francesco sorry im on my phone and have not been running giraph on
>>> yarn for some time but if i had to guess this looks like a version
>>> compatibility issue with the version of yarn the giraph is built against
>>> compared to what you ran on ec2 or perhaps the version of protobuf dep in
>>> the giraph build and the yarn cluster? either way id try an earlier version
>>> of yarn/hadoop2 for ec2 and your giraph build, as giraph hasnt been keeping
>>> up with all the yarn side changes.
>>> alternate theory is i have seen startup argument and config errors show
>>> up with ambiguous traces similar to that. not able to review your conf file
>>> from phone but you might review configs and args against giraph and yarn
>>> code directly to ensure all those are good since docs might be outdated on
>>> some of those.
>>> Good luck!
>>> On Sat, Jul 28, 2018, 9:42 PM Francesco Sclano <
>>>> wrote:
>>>> Hi,
>>>> I'm using giraph-1.2 for my master thesis in computer science. I
>>>> developed in giraph the calculus of 4-profiles like
>>>> I succesfully configured and runned giraph-1.2 and hadoop-2.5.1 in
>>>> pseudo distributed mode on my local pc with this configuration
>>>> <>.
>>>> In pseudo distributed mode I launch my giraph program with following
>>>> parameters:
>>>> giraph MY_JAR.jar SUPERSTEP0_CLASS -ca giraph.master.observers=MY_OBSERVER_CLASS
>>>> -mc MY_MASTER_CLASS -eif MY_CUSTOM_INPUT_FORMAT -eip input.txt -vof
>>>> -op output -w
>>>> 1 -ca giraph.SplitMasterWorker=false -ca io.edge.reverse.duplicator=
>>>> true
>>>> Then I tryed to use amazon ec2 with a simple cluster of 3 nodes with
>>>> this other configuration <> but I obtain the
>>>> error below. I also re-tryed pseudo-distributed mode in amazon ec2 with
>>>> only 1 node and I obtain the same error below, but on my local pc it
>>>> works!!!
>>>> I'm using ubuntu 16.04 both in local and in ec2.
>>>> Obviously I tried an hadoop example on my ec2 cluster and it works.
>>>> Please help me because I'm blocked with my master this for this error!
>>>> I red many forums about this error and I checked many times that I have
>>>> only protobuf-java-2.5.0.jar inside hadoop-2.5.2 and giraph-1.2 both in ec2
>>>> and in my local pc.
>>>> FATAL [main] Error
>>>> starting MRAppMaster
>>>> java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto
>>>> overrides final method getUnknownFields.()Lcom/google/protobuf/
>>>> UnknownFieldSet;
>>>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>>>         at java.lang.ClassLoader.defineClass(
>>>>         at
>>>>         at
>>>>         at$100(
>>>>         at$
>>>>         at$
>>>>         at Method)
>>>>         at
>>>>         at java.lang.ClassLoader.loadClass(
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(
>>>>         at java.lang.ClassLoader.loadClass(
>>>>         at java.lang.Class.getDeclaredConstructors0(Native Method)
>>>>         at java.lang.Class.privateGetDeclaredConstructors
>>>> (
>>>>         at java.lang.Class.getConstructor0(
>>>>         at java.lang.Class.getConstructor(
>>>>         at org.apache.hadoop.yarn.factories.impl.pb.
>>>> RecordFactoryPBImpl.newRecordInstance(
>>>>         at org.apache.hadoop.yarn.util.Records.newRecord(Records.
>>>> java:36)
>>>>         at org.apache.hadoop.yarn.api.records.ApplicationId.
>>>> newInstance(
>>>>         at org.apache.hadoop.yarn.util.ConverterUtils.
>>>> toApplicationAttemptId(
>>>>         at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(
>>>>         at
>>>> INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
>>>> Links of configurafion files works until 4 August 2018 but I can
>>>> re-send them.
>>>> Many Thanks
>>>> Best Regards
>>>> --
>>>> Francesco Sclano
>> --
>> Francesco Sclano

Francesco Sclano

View raw message