Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0FD7B186E6 for ; Tue, 15 Dec 2015 17:14:58 +0000 (UTC) Received: (qmail 55853 invoked by uid 500); 15 Dec 2015 17:14:52 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 55758 invoked by uid 500); 15 Dec 2015 17:14:52 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 55747 invoked by uid 99); 15 Dec 2015 17:14:52 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Dec 2015 17:14:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id EBF01C0D43 for ; Tue, 15 Dec 2015 17:14:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.346 X-Spam-Level: ** X-Spam-Status: No, score=2.346 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RP_MATCHES_RCVD=-0.554, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id JkspVKv4aHQI for ; Tue, 15 Dec 2015 17:14:37 +0000 (UTC) Received: from nm21.bullet.mail.bf1.yahoo.com (nm21.bullet.mail.bf1.yahoo.com [98.139.212.180]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 0A939206E3 for ; Tue, 15 Dec 2015 17:14:36 +0000 (UTC) Received: from [98.139.215.140] by nm21.bullet.mail.bf1.yahoo.com with NNFMP; 15 Dec 2015 17:14:30 -0000 Received: from [68.142.230.77] by tm11.bullet.mail.bf1.yahoo.com with NNFMP; 15 Dec 2015 17:14:30 -0000 Received: from [127.0.0.1] by smtp234.mail.bf1.yahoo.com with NNFMP; 15 Dec 2015 17:14:30 -0000 X-Yahoo-Newman-Id: 670155.21776.bm@smtp234.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 8X3JcroVM1kaG6oILA7mW9gqR.O2pe.FwyKMj_IY2Lw0ed3 kyUtFtE2gTT3ZmTmmCCoS6xkHq2zqyBs.Q70BiL75nig5iRipClO2SKhAbj3 .yO3.wPS7_nQOg0EVJ4AG6.8EuRLoljqn_k0eA_b_1XN_ZThtSZC5.260_TD xxANiuKvRILiJHxJtvCXrYtVz_Oy1wdxyEKEUDcH6vV0.78WmUM8Eu_JOoa_ hTI.pRSVXr0IkCkL_GN9BtZYf2c4OfuV1b0wSk0mRoj8EUL8WaRA3htOudMI 8qdtimNOG5i_LkhhFLX4ALdrTTy25h_lHM.4Grmxwmw32bxl2hmgDRUcRtlU 5pPMJ3HTAOqUbeUz1DYIE_cON8tkmXltRr4re8xiRF20c3wqsZg0eFtzepcn .NHwnc7sy.3YbhWHbHY4I9HCraj4O3kt3qE_5CfH95Y7W8p1Uby6s5WPRtcc zYT.hOOUGRE2IgAbgmt81FqmT6ADZ1QbCOqbPeCOOZTzHlqun6zojxoj5a3Z BuzA- X-Yahoo-SMTP: IS3.70aswBCGWA9DaPXCS.rX2K7W2dwjrA-- Subject: Re: GridMix doesn't run ClassNotFoundException Rumen To: Wellington Chevreuil References: <566E925E.6070408@yahoo.com> <56700D2A.7080608@yahoo.com> Cc: user@hadoop.apache.org From: siscia Message-ID: <56704A74.4080003@yahoo.com> Date: Tue, 15 Dec 2015 18:14:28 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------040307000007040408090805" --------------040307000007040408090805 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, It looks like everything is in place but still doesn't work... Let me recap, I am running in pseudo distributed mode. I start the system with this sequence of command. cd hadoop-2.6.1 sbin/start-dfs.sh sbin/start-yarn.sh sbin/hadoop-daemon.sh start datanode sbin/yarn-daemon.sh start nodemanager Now the single node of the system is healthy and everything looks cool. jps give me this: simo@simo:~/hadoop-2.6.1$ jps 10107 Jps 4114 NodeManager 3533 SecondaryNameNode 3344 NameNode 4014 DataNode 3737 ResourceManager I check which user is running the NodeManager simo@simo:~/hadoop-2.6.1$ ps u 4114 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND simo 4114 1.9 8.7 1764708 346188 pts/3 Sl 17:37 0:39 /usr/bin/java -Dproc_nodemanager -Xmx1000m -server -Dhadoop.log.dir=/home/simo/hadoop I am running the nodemanager. Let me check the permission of the rumen jar simo@simo:~/hadoop-2.6.1$ ls -l share/hadoop/tools/lib/ | grep rumen -rw-r--r-- 1 simo simo 278018 set 16 23:18 hadoop-rumen-2.6.1.jar I can read the rumen jar. I don't have a lot of ideas of what I could be doing wrong... You have any thoughts ? Thanks for your time, Simone On 15/12/2015 16:27, Wellington Chevreuil wrote: > > Hi, > > Since it's on yarn class path, it should already be on Nodemanager or > the jobs path. You can check the Nodemanager process itself, with a ps > command. > > Can you also check the file permissions for this jar? Make sure it's > readable by the user running Nodemanager process. > > Em 15/12/2015 12:53, "siscia" > escreveu: > > Hi Wellington, > > I am missing a passage, if I check the classpath of both hadoop > and yarn they should include hadoop-rumen-2.6.1.jar > > simo@simo:~/hadoop-2.6.1$ > bin/hadoop classpath > /home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/share/hadoop/common/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/common/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/*:*/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/*:*/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/**:/usr/lib/jvm/java-7-openjdk-amd64/lib/:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar:HADOOP_CLASSPATH > > simo@simo:~/hadoop-2.6.1$ > bin/yarn classpath > /home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/share/hadoop/common/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/common/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/*:/*home/simo/hadoop-2.6.1/share/hadoop/tools/lib/*:*/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/**:/usr/lib/jvm/java-7-openjdk-amd64/lib/:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar:HADOOP_CLASSPATH:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/* > > > Given that > > simo@simo:~*/hadoop-2.6.1/share/hadoop/tools/lib*$ ls > hadoop-extras-2.6.1.jar servlet-api-2.5.jar > *hadoop-gridmix-2.6.1.jar* snappy-java-1.0.4.1.jar > hadoop-openstack-2.6.1.jar stax-api-1.0-2.jar > hadoop-rumen-2.6.1.jar xmlenc-0.52.jar > hadoop-sls-2.6.1.jar xz-1.0.jar > hadoop-streaming-2.6.1.jar zookeeper-3.4.6.jar > > Am I doing something wrong ? How do I check the classpath of the > NodeManager ? > > Greets > > Simone > > On 15/12/2015 12:45, Wellington Chevreuil wrote: >> Hi Simone, >> >> You should make sure to include hadoop-rumen-2.6.0.jar on the >> classpath for the Nodemanagers, or include it on the classpath of >> your job. >> >>> On 14 Dec 2015, at 09:56, siscia >> > wrote: >>> >>> Hello folks, >>> >>> I am trying to run a simulation with GridMix but the simulation >>> fails every time. >>> >>> If I inspect the log I find the error with looks like this: >>> >>> "Error: >>> *java.lang.ClassNotFoundException: >>> org.apache.hadoop.tools.rumen.ResourceUsageMetrics\n\tat* >>> >>> java.net.URLClassLoader$1.run(URLClassLoader.java:366)\n\tat >>> java.net.URLClassLoader$1.run(URLClassLoader.java:355)\n\tat >>> java.security.AccessController.doPrivileged(Native >>> Method)\n\tat >>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)\n\tat >>> java.lang.ClassLoader.loadClass(ClassLoader.java:425)\n\tat >>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)\n\tat >>> java.lang.ClassLoader.loadClass(ClassLoader.java:358)\n\tat >>> java.lang.Class.getDeclaredConstructors0(Native >>> Method)\n\tat >>> java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)\n\tat >>> java.lang.Class.getConstructor0(Class.java:2895)\n\tat >>> java.lang.Class.getDeclaredConstructor(Class.java:2066)\n\tat org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)\n\tat >>> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:66)\n\tat >>> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)\n\tat >>> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)\n\tat >>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)\n\tat >>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\n\tat >>> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)\n\tat >>> java.security.AccessController.doPrivileged(Native >>> Method)\n\tat >>> javax.security.auth.Subject.doAs(Subject.java:415)\n\tat >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)\n\tat >>> org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\n", >>> >>> The error is pretty clear, but I have no idea how to fix it. >>> >>> I run GridMix by command line with: >>> >>> bin/hadoop jar >>> share/hadoop/tools/lib/hadoop-gridmix-2.6.1.jar iopath >>> trace.json >>> >>> Any hits ? >>> >>> Cheers :) >>> >>> Simone >> > --------------040307000007040408090805 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Hi,

It looks like everything is in place but still doesn't work...

Let me recap, I am running in pseudo distributed mode.

I start the system with this sequence of command.
cd hadoop-2.6.1
sbin/start-dfs.sh
sbin/start-yarn.sh
sbin/hadoop-daemon.sh start datanode
sbin/yarn-daemon.sh start nodemanager

Now the single node of the system is healthy and everything looks cool.

jps give me this:
simo@simo:~/hadoop-2.6.1$ jps
10107 Jps
4114 NodeManager
3533 SecondaryNameNode
3344 NameNode
4014 DataNode
3737 ResourceManager

I check which user is running the NodeManager
simo@simo:~/hadoop-2.6.1$ ps u 4114
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
simo      4114  1.9  8.7 1764708 346188 pts/3  Sl   17:37   0:39 /usr/bin/java -Dproc_nodemanager -Xmx1000m -server -Dhadoop.log.dir=/home/simo/hadoop

I am running the nodemanager.

Let me check the permission of the rumen jar
simo@simo:~/hadoop-2.6.1$ ls -l share/hadoop/tools/lib/ | grep rumen
-rw-r--r-- 1 simo simo   278018 set 16 23:18 hadoop-rumen-2.6.1.jar

I can read the rumen jar.

I don't have a lot of ideas of what I could be doing wrong...

You have any thoughts ?

Thanks for your time,

Simone


On 15/12/2015 16:27, Wellington Chevreuil wrote:

Hi,

Since it's on yarn class path, it should already be on Nodemanager or the jobs path. You can check the Nodemanager process itself, with a ps command.

Can you also check the file permissions for this jar? Make sure it's readable by the user running Nodemanager process.

Em 15/12/2015 12:53, "siscia" <sisciamirror@yahoo.com> escreveu:
Hi Wellington,

I am missing a passage, if I check the classpath of both hadoop and yarn they should include hadoop-rumen-2.6.1.jar

simo@simo:~/hadoop-2.6.1$ bin/hadoop classpath
/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/share/hadoop/common/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/common/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/*:/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/:/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/*:/usr/lib/jvm/java-7-openjdk-amd64/lib/:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar:HADOOP_CLASSPATH

simo@simo:~/hadoop-2.6.1$ bin/yarn classpath
/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/share/hadoop/common/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/common/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/*:/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/:/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/*:/usr/lib/jvm/java-7-openjdk-amd64/lib/:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar:HADOOP_CLASSPATH:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*

Given that
simo@simo:~/hadoop-2.6.1/share/hadoop/tools/lib$ ls
hadoop-extras-2.6.1.jar                servlet-api-2.5.jar
hadoop-gridmix-2.6.1.jar               snappy-java-1.0.4.1.jar
hadoop-openstack-2.6.1.jar             stax-api-1.0-2.jar
hadoop-rumen-2.6.1.jar                 xmlenc-0.52.jar
hadoop-sls-2.6.1.jar                   xz-1.0.jar
hadoop-streaming-2.6.1.jar             zookeeper-3.4.6.jar

Am I doing something wrong ? How do I check the classpath of the NodeManager ?

Greets

Simone

On 15/12/2015 12:45, Wellington Chevreuil wrote:
Hi Simone,

You should make sure to include hadoop-rumen-2.6.0.jar on the classpath for the Nodemanagers, or include it on the classpath of your job.

On 14 Dec 2015, at 09:56, siscia <sisciamirror@yahoo.com.INVALID> wrote:

Hello folks,

I am trying to run a simulation with GridMix but the simulation fails every time.

If I inspect the log I find the error with looks like this:
"Error:
java.lang.ClassNotFoundException: org.apache.hadoop.tools.rumen.ResourceUsageMetrics\n\tat

java.net.URLClassLoader$1.run(URLClassLoader.java:366)\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:355)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:354)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:425)\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:358)\n\tat java.lang.Class.getDeclaredConstructors0(Native Method)\n\tat java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)\n\tat java.lang.Class.getConstructor0(Class.java:2895)\n\tat java.lang.Class.getDeclaredConstructor(Class.java:2066)\n\tat org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)\n\tat org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:66)\n\tat org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)\n\tat org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)\n\tat org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)\n\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\n",

The error is pretty clear, but I have no idea how to fix it.

I run GridMix by command line with:
bin/hadoop jar share/hadoop/tools/lib/hadoop-gridmix-2.6.1.jar iopath trace.json

Any hits ?

Cheers :)

Simone



--------------040307000007040408090805--