giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francesco Sclano <sclano.france...@gmail.com>
Subject Re: I'm stucked with this error: Error starting MRAppMaster java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
Date Mon, 06 Aug 2018 09:55:15 GMT
Hi Eli,
first of all thank you for your support and your time!
I tryed to use giraph-dist-1.2.0-hadoop2-bin.tar.gz with hadoop-2.3.0 but I
got always the same error. I read all previous posts but I didn't find any
solution. I found this post about my error
https://blog.woopi.org/wordpress/?p=137 but I didn't find any occurrence of
"google protocol buffer jar with version 2.4.1" in my environment. Maybe
giraph-dist-1.2.0-bin.tar.gz was wrongly build with 2.4.1 dependency in
place of 2.5.0?! I tryed to download giraph-dist-1.2.0-hadoop2-src.zip and
build it but the build fails on giraph-core. I used "mvn -Phadoop_2
-DskipTests clean install". Could you please provide me the right mvn
command or the right mvn configuration to be used?

*Hence I tryed to use giraph-dist-1.2.0-bin.tar.gz with hadoop-1.2.1 and
this time it properly works: I succeed to use giraph in a ec2 cluster with
this input graph http://snap.stanford.edu/data/p2p-Gnutella08.html
<http://snap.stanford.edu/data/p2p-Gnutella08.html> (for beginning 1
namenode and 2 datenodes with free micro tier - ubuntu 16.04). I didn't
change anything in my giraph parameters!!!* Anyway I'd prefer to use
giraph-1.2 with hadoop-2.

I found this
https://stackoverflow.com/questions/33728752/building-giraph-with-hadoop
and also the documentation below in https://github.com/apache/giraph, but
I'm still a bit confused. In order to build giraph1.2 with hadoop-2.5.1 I
have to use "mvn -Phadoop_2 clean package -DskipTests" or I have to use
"mvn -Phadoop_yarn -Dhadoop.version=2.5.1 clean package -DskipTests". Why
both builds fail on giraph-core? Is giraph-1.2 compatible with hadoop-2.5.1
without using yarn? How hadoop-2.5.1 can be configured without using yarn?
Thanks

- Apache Hadoop 2 (latest version: 2.5.1)

  This is the latest version of Hadoop 2 (supporting YARN in addition
  to MapReduce) Giraph could use. You may tell maven to use this version
  with "mvn -Phadoop_2 <goals>".





2018-08-01 12:11 GMT+02:00 Eli Reisman <apache.mailbox@gmail.com>:

> hi francesco,
>
> I think in the mailing list you might find some tips for running on hadoop
> 2.5.x but IIRC hadoop 2.3.x plus appropriate Giraph build works. Again,
> given the error msg I'd also carefully review your command line options and
> config file just in case.
>
> Also for context as far as I know no one is actively maintaining Giraph on
> YARN compatibility for newer versions right now so taking a look at the
> mailing list backscroll will probably yield the best info on which
> combinations work best
>
> On Mon, Jul 30, 2018, 11:52 AM Francesco Sclano <
> sclano.francesco@gmail.com> wrote:
>
>> Hi Eli,
>> ok, thanks for you suggestion. I'll try it and let you know. In order to
>> be sure, before try to use an earlier version of yarn/hadoop2, I'm using
>> giraph-dist-1.2.0-hadoop2-bin.tar.gz. Does this giraph version
>> officially support hadoop-2.5.1 with yarn? If not how I have to configure
>> hadoop-2.5.1? Thanks
>>
>> 2018-07-29 9:47 GMT+02:00 Eli Reisman <apache.mailbox@gmail.com>:
>>
>>> hey francesco sorry im on my phone and have not been running giraph on
>>> yarn for some time but if i had to guess this looks like a version
>>> compatibility issue with the version of yarn the giraph is built against
>>> compared to what you ran on ec2 or perhaps the version of protobuf dep in
>>> the giraph build and the yarn cluster? either way id try an earlier version
>>> of yarn/hadoop2 for ec2 and your giraph build, as giraph hasnt been keeping
>>> up with all the yarn side changes.
>>>
>>> alternate theory is i have seen startup argument and config errors show
>>> up with ambiguous traces similar to that. not able to review your conf file
>>> from phone but you might review configs and args against giraph and yarn
>>> code directly to ensure all those are good since docs might be outdated on
>>> some of those.
>>>
>>> Good luck!
>>>
>>> On Sat, Jul 28, 2018, 9:42 PM Francesco Sclano <
>>> sclano.francesco@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I'm using giraph-1.2 for my master thesis in computer science. I
>>>> developed in giraph the calculus of 4-profiles like
>>>> eelenberg.github.io/Elenberg4profileWWW16.pdf
>>>> I succesfully configured and runned giraph-1.2 and hadoop-2.5.1 in
>>>> pseudo distributed mode on my local pc with this configuration
>>>> <https://we.tl/dB0PmGhUPX>.
>>>>
>>>> In pseudo distributed mode I launch my giraph program with following
>>>> parameters:
>>>> giraph MY_JAR.jar SUPERSTEP0_CLASS -ca giraph.master.observers=MY_OBSERVER_CLASS
>>>> -mc MY_MASTER_CLASS -eif MY_CUSTOM_INPUT_FORMAT -eip input.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op output -w
>>>> 1 -ca giraph.SplitMasterWorker=false -ca io.edge.reverse.duplicator=
>>>> true
>>>>
>>>> Then I tryed to use amazon ec2 with a simple cluster of 3 nodes with
>>>> this other configuration <https://we.tl/Gbv9q0nQ7h> but I obtain the
>>>> error below. I also re-tryed pseudo-distributed mode in amazon ec2 with
>>>> only 1 node and I obtain the same error below, but on my local pc it
>>>> works!!!
>>>> I'm using ubuntu 16.04 both in local and in ec2.
>>>>
>>>> Obviously I tried an hadoop example on my ec2 cluster and it works.
>>>> Please help me because I'm blocked with my master this for this error!
>>>> I red many forums about this error and I checked many times that I have
>>>> only protobuf-java-2.5.0.jar inside hadoop-2.5.2 and giraph-1.2 both in ec2
>>>> and in my local pc.
>>>>
>>>>
>>>>
>>>> FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error
>>>> starting MRAppMaster
>>>> java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto
>>>> overrides final method getUnknownFields.()Lcom/google/protobuf/
>>>> UnknownFieldSet;
>>>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>>         at java.security.SecureClassLoader.defineClass(
>>>> SecureClassLoader.java:142)
>>>>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(
>>>> Launcher.java:349)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>         at java.lang.Class.getDeclaredConstructors0(Native Method)
>>>>         at java.lang.Class.privateGetDeclaredConstructors
>>>> (Class.java:2671)
>>>>         at java.lang.Class.getConstructor0(Class.java:3075)
>>>>         at java.lang.Class.getConstructor(Class.java:1825)
>>>>         at org.apache.hadoop.yarn.factories.impl.pb.
>>>> RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:62)
>>>>         at org.apache.hadoop.yarn.util.Records.newRecord(Records.
>>>> java:36)
>>>>         at org.apache.hadoop.yarn.api.records.ApplicationId.
>>>> newInstance(ApplicationId.java:49)
>>>>         at org.apache.hadoop.yarn.util.ConverterUtils.
>>>> toApplicationAttemptId(ConverterUtils.java:137)
>>>>         at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(
>>>> ConverterUtils.java:177)
>>>>         at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
>>>> MRAppMaster.java:1391)
>>>>
>>>> INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
>>>>
>>>>
>>>> Links of configurafion files works until 4 August 2018 but I can
>>>> re-send them.
>>>>
>>>> Many Thanks
>>>>
>>>> Best Regards
>>>>
>>>> --
>>>> Francesco Sclano
>>>>
>>>
>>
>>
>> --
>> Francesco Sclano
>>
>


-- 
Francesco Sclano

Mime
View raw message