From user-return-3654-archive-asf-public=cust-asf.ponee.io@giraph.apache.org Wed Aug 1 12:12:01 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 78D3A180634 for ; Wed, 1 Aug 2018 12:12:00 +0200 (CEST) Received: (qmail 38819 invoked by uid 500); 1 Aug 2018 10:11:59 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 38808 invoked by uid 99); 1 Aug 2018 10:11:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Aug 2018 10:11:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1076CC0B71 for ; Wed, 1 Aug 2018 10:11:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.87 X-Spam-Level: * X-Spam-Status: No, score=1.87 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 2SMnCmwHVJkB for ; Wed, 1 Aug 2018 10:11:57 +0000 (UTC) Received: from mail-yw0-f195.google.com (mail-yw0-f195.google.com [209.85.161.195]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E33BE5F1B9 for ; Wed, 1 Aug 2018 10:11:56 +0000 (UTC) Received: by mail-yw0-f195.google.com with SMTP id w76-v6so6960869ywg.4 for ; Wed, 01 Aug 2018 03:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=PnN0abpj7vHQWc4tnoNU4+qNsxMXg4xmKn6iDvMHUjE=; b=H7HCDnqjPx80l2gVlsaGQrQkfFZ2d2g0ue82Hwv88Fxsv7pz3BU7FRBG0PSWWechjy fq5oNwAIexM5EH8uoT6ARjTX4yG9IS5IgFk1VJWe7cxQ7Tij7+ryjlGrZ8SnPy+1DAKS LvRLpIseaDnaF0pQYYdxMkQF/o5/sr6Yl/r9z54lzUpDHwYYXNxeiAq/q71C72MRH9Ti MnYq7FicANEdnDJXY1tw9whAmQgwhKfYxHvUSDzy0fO88ASjMVkg7RgDuXuQzC8AYiny AXb30pJ3I/Nbvc8e0Zfm0O1SySvAH/mz1TTcgrs+sLOG9hftP6aUWnGhKc81oHVSj7Sm W3dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=PnN0abpj7vHQWc4tnoNU4+qNsxMXg4xmKn6iDvMHUjE=; b=Ok76tfy4ElLVBPc2K1ya/TZ0Gn1d/dOe+OfcuCqseQK2lsv7BRV+xeiAh5xGiNxptV i00arOlaU/ZX0qesqP0qfghmbi8El6P+hd9zmsB+38OR2UQWrS2BXs9aWX/kKCf1Pqwm MnxFy6yfUNQ2QuIsodpKojtyhLPo4n2X8nQT73nCwGUDEFCPFO2OnJmyZr4Uoj1SbzOT sw6tPgkEaC3oMgaxT9KijgLxJZsYryaGmnltKcNJvLTii3VwFap88LBRi1b27FSgIQWC tC2ubagUpcZGg3ajAIcHjdje2/kIlKSyX8wuQNbdwEzAfDUi7Xk8XJf6xKff/QNJrr7E cXQg== X-Gm-Message-State: AOUpUlEL1pkJfLBuInR3FE2u+lXvv23ao3p2WiefQF+lMBKxP08AFp5Z 53C9JzYJnxqXfDj5niJf09ZFRF/SSISxjyF/yS4qTrtM X-Google-Smtp-Source: AAOMgpeuwyGmR8ylYaUu1Tf/V2zHjAJhqccFVsYQZLXoF4l0fbgURnfItgo0l+/9OGo3InDAjUz3105qGLKUvp4hEKQ= X-Received: by 2002:a81:810:: with SMTP id 16-v6mr12807700ywi.389.1533118315693; Wed, 01 Aug 2018 03:11:55 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Eli Reisman Date: Wed, 1 Aug 2018 13:11:43 +0300 Message-ID: Subject: Re: I'm stucked with this error: Error starting MRAppMaster java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; To: user@giraph.apache.org Content-Type: multipart/alternative; boundary="0000000000009b0b4d05725cec32" --0000000000009b0b4d05725cec32 Content-Type: text/plain; charset="UTF-8" hi francesco, I think in the mailing list you might find some tips for running on hadoop 2.5.x but IIRC hadoop 2.3.x plus appropriate Giraph build works. Again, given the error msg I'd also carefully review your command line options and config file just in case. Also for context as far as I know no one is actively maintaining Giraph on YARN compatibility for newer versions right now so taking a look at the mailing list backscroll will probably yield the best info on which combinations work best On Mon, Jul 30, 2018, 11:52 AM Francesco Sclano wrote: > Hi Eli, > ok, thanks for you suggestion. I'll try it and let you know. In order to > be sure, before try to use an earlier version of yarn/hadoop2, I'm using > giraph-dist-1.2.0-hadoop2-bin.tar.gz. Does this giraph version officially > support hadoop-2.5.1 with yarn? If not how I have to configure > hadoop-2.5.1? Thanks > > 2018-07-29 9:47 GMT+02:00 Eli Reisman : > >> hey francesco sorry im on my phone and have not been running giraph on >> yarn for some time but if i had to guess this looks like a version >> compatibility issue with the version of yarn the giraph is built against >> compared to what you ran on ec2 or perhaps the version of protobuf dep in >> the giraph build and the yarn cluster? either way id try an earlier version >> of yarn/hadoop2 for ec2 and your giraph build, as giraph hasnt been keeping >> up with all the yarn side changes. >> >> alternate theory is i have seen startup argument and config errors show >> up with ambiguous traces similar to that. not able to review your conf file >> from phone but you might review configs and args against giraph and yarn >> code directly to ensure all those are good since docs might be outdated on >> some of those. >> >> Good luck! >> >> On Sat, Jul 28, 2018, 9:42 PM Francesco Sclano < >> sclano.francesco@gmail.com> wrote: >> >>> Hi, >>> I'm using giraph-1.2 for my master thesis in computer science. I >>> developed in giraph the calculus of 4-profiles like >>> eelenberg.github.io/Elenberg4profileWWW16.pdf >>> I succesfully configured and runned giraph-1.2 and hadoop-2.5.1 in >>> pseudo distributed mode on my local pc with this configuration >>> . >>> >>> In pseudo distributed mode I launch my giraph program with following >>> parameters: >>> giraph MY_JAR.jar SUPERSTEP0_CLASS -ca >>> giraph.master.observers=MY_OBSERVER_CLASS -mc MY_MASTER_CLASS -eif >>> MY_CUSTOM_INPUT_FORMAT -eip input.txt -vof >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op output -w 1 >>> -ca giraph.SplitMasterWorker=false -ca io.edge.reverse.duplicator=true >>> >>> Then I tryed to use amazon ec2 with a simple cluster of 3 nodes with >>> this other configuration but I obtain the >>> error below. I also re-tryed pseudo-distributed mode in amazon ec2 with >>> only 1 node and I obtain the same error below, but on my local pc it >>> works!!! >>> I'm using ubuntu 16.04 both in local and in ec2. >>> >>> Obviously I tried an hadoop example on my ec2 cluster and it works. >>> Please help me because I'm blocked with my master this for this error! I >>> red many forums about this error and I checked many times that I have only >>> protobuf-java-2.5.0.jar inside hadoop-2.5.2 and giraph-1.2 both in ec2 and >>> in my local pc. >>> >>> >>> >>> FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error >>> starting MRAppMaster >>> java.lang.VerifyError: class >>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final >>> method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; >>> at java.lang.ClassLoader.defineClass1(Native Method) >>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) >>> at >>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) >>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> at java.lang.Class.getDeclaredConstructors0(Native Method) >>> at >>> java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) >>> at java.lang.Class.getConstructor0(Class.java:3075) >>> at java.lang.Class.getConstructor(Class.java:1825) >>> at >>> org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:62) >>> at org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36) >>> at >>> org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49) >>> at >>> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) >>> at >>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) >>> at >>> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1391) >>> >>> INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1 >>> >>> >>> Links of configurafion files works until 4 August 2018 but I can re-send >>> them. >>> >>> Many Thanks >>> >>> Best Regards >>> >>> -- >>> Francesco Sclano >>> >> > > > -- > Francesco Sclano > --0000000000009b0b4d05725cec32 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
hi francesco,

I think in the mailing list you might find some tips for running on hadoo= p 2.5.x but IIRC hadoop 2.3.x plus appropriate Giraph build works. Again, g= iven the error msg I'd also carefully review your command line options = and config file just in case.

Also for context as far as I know no one is actively maintaining Gira= ph on YARN compatibility for newer versions right now so taking a look at t= he mailing list backscroll will probably yield the best info on which combi= nations work best

On Mon, Jul 30, 2018, 11:52 AM Francesco Sclano <sclano.francesco@gmail.com> wrote:
Hi Eli,
ok, thanks for = you suggestion. I'll try it and let you know. In order to be sure, befo= re try to use an earlier version of yarn/hadoop2, I'm using giraph-dist= -1.2.0-hadoop2-bin.tar.gz. Does this giraph version officially support hado= op-2.5.1 with yarn? If not how I have to configure hadoop-2.5.1? Thanks
=

2018-07-29 = 9:47 GMT+02:00 Eli Reisman <apache.mailbox@gmail= .com>:
hey francesco sorry im on my phone and have not been running giraph on y= arn for some time but if i had to guess this looks like a version compatibi= lity issue with the version of yarn the giraph is built against compared to= what you ran on ec2 or perhaps the version of protobuf dep in the giraph b= uild and the yarn cluster? either way id try an earlier version of yarn/had= oop2 for ec2 and your giraph build, as giraph hasnt been keeping up with al= l the yarn side changes.

alternate theory is i have seen startup argument and config errors show up= with ambiguous traces similar to that. not able to review your conf file f= rom phone but you might review configs and args against giraph and yarn cod= e directly to ensure all those are good since docs might be outdated on som= e of those.

Good luck!

On Sat, Jul 28, 2018, 9= :42 PM Francesco Sclano <sclano.francesco@gmail.com> wrot= e:
Hi,
<= div>I'm using giraph-1.2 for my master thesis in computer science. I de= veloped in giraph the calculus of 4-profiles like=20 eelenberg.github.io/E= lenberg4profileWWW16.pdf
I succesfully configured and runned giraph-1.2 and hadoop-2.= 5.1 in pseudo distributed mode on my local pc with this configura= tion.

In pseudo distributed mode I launch my g= iraph program with following parameters:
giraph MY_JAR.jar SUPERS= TEP0_CLASS -ca giraph.master.observers=3DMY_OBSERVER_CLASS -mc MY_MASTER_CL= ASS -eif MY_CUSTOM_INPUT_FORMAT -eip input.txt -vof org.apache.giraph.io.fo= rmats.IdWithValueTextOutputFormat -op output -w 1 -ca giraph.SplitMasterWor= ker=3Dfalse -ca io.edge.reverse.duplicator=3Dtrue

= Then I tryed to use amazon ec2 with a simple cluster of 3 nodes with this <= a href=3D"https://we.tl/Gbv9q0nQ7h" rel=3D"noreferrer noreferrer" target=3D= "_blank">other configuration but I obtain the error below. I also re-tr= yed pseudo-distributed mode in amazon ec2 with only 1 node and I obtain the= same error below, but on my local pc it works!!!
I'm us= ing ubuntu 16.04 both in local and in ec2.

Obvious= ly I tried an hadoop example on my ec2 cluster and it works.
= Please help me because I'm blocked with my master this for this error! = I red many forums about this error and I checked many times that I have onl= y protobuf-java-2.5.0.jar inside hadoop-2.5.2 and giraph-1.2 both in ec2 an= d in my local pc.



FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starti= ng MRAppMaster
java.lang.VerifyError: class org.apache.hadoop.yarn.proto= .YarnProtos$ApplicationIdProto overrides final method getUnknownFields.()Lc= om/google/protobuf/UnknownFieldSet;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 at java.lang.ClassLoader.defineClass1(Native Method)
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.ClassLoader.defineClass(ClassLo= ader.java:763)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.securi= ty.SecureClassLoader.defineClass(SecureClassLoader.java:142)
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader.defineClass(UR= LClassLoader.java:467)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at jav= a.net.URLClassLoader.access$100(URLClassLoader.java:73)
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader$1.run(URLClassLoader= .java:368)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.net.URLCla= ssLoader$1.run(URLClassLoader.java:362)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 at java.security.AccessController.doPrivileged(Native Method)<= br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader.fi= ndClass(URLClassLoader.java:361)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.misc.Launcher$AppClassLoader.loadC= lass(Launcher.java:349)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at ja= va.lang.ClassLoader.loadClass(ClassLoader.java:357)
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Class.getDeclaredConstructors0(Native= Method)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Class.p= rivateGetDeclaredConstructors(Class.java:2671)
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 at java.lang.Class.getConstructor0(Class.java:3075)
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Class.getConstructo= r(Class.java:1825)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apa= che.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(Rec= ordFactoryPBImpl.java:62)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at = org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36)
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.api.records.App= licationId.newInstance(ApplicationId.java:49)
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.util.ConverterUtils.toApplicat= ionAttemptId(ConverterUtils.java:137)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(Conve= rterUtils.java:177)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.ap= ache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1391)

INFO [main] org.apache.hadoop.util.ExitUtil: Exiting= with status 1


Links of configu= rafion files works until 4 August 2018 but I can re-send them.

Many Thanks

Best Regards

--
Francesco Sclano



--
Francesco S= clano
--0000000000009b0b4d05725cec32--