hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siscia <sisciamir...@yahoo.com.INVALID>
Subject Re: GridMix doesn't run ClassNotFoundException Rumen
Date Tue, 15 Dec 2015 17:14:28 GMT
Hi,

It looks like everything is in place but still doesn't work...

Let me recap, I am running in pseudo distributed mode.

I start the system with this sequence of command.

    cd hadoop-2.6.1
    sbin/start-dfs.sh
    sbin/start-yarn.sh
    sbin/hadoop-daemon.sh start datanode
    sbin/yarn-daemon.sh start nodemanager


Now the single node of the system is healthy and everything looks cool.

jps give me this:

    simo@simo:~/hadoop-2.6.1$ jps
    10107 Jps
    4114 NodeManager
    3533 SecondaryNameNode
    3344 NameNode
    4014 DataNode
    3737 ResourceManager


I check which user is running the NodeManager

    simo@simo:~/hadoop-2.6.1$ ps u 4114
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    simo      4114  1.9  8.7 1764708 346188 pts/3  Sl 17:37   0:39
    /usr/bin/java -Dproc_nodemanager -Xmx1000m -server
    -Dhadoop.log.dir=/home/simo/hadoop


I am running the nodemanager.

Let me check the permission of the rumen jar

    simo@simo:~/hadoop-2.6.1$ ls -l share/hadoop/tools/lib/ | grep rumen
    -rw-r--r-- 1 simo simo   278018 set 16 23:18 hadoop-rumen-2.6.1.jar


I can read the rumen jar.

I don't have a lot of ideas of what I could be doing wrong...

You have any thoughts ?

Thanks for your time,

Simone


On 15/12/2015 16:27, Wellington Chevreuil wrote:
>
> Hi,
>
> Since it's on yarn class path, it should already be on Nodemanager or 
> the jobs path. You can check the Nodemanager process itself, with a ps 
> command.
>
> Can you also check the file permissions for this jar? Make sure it's 
> readable by the user running Nodemanager process.
>
> Em 15/12/2015 12:53, "siscia" <sisciamirror@yahoo.com 
> <mailto:sisciamirror@yahoo.com>> escreveu:
>
>     Hi Wellington,
>
>     I am missing a passage, if I check the classpath of both hadoop
>     and yarn they should include hadoop-rumen-2.6.1.jar
>
>         simo@simo:~/hadoop-2.6.1$ <mailto:simo@simo:%7E/hadoop-2.6.1$>
>         bin/hadoop classpath
>         /home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/share/hadoop/common/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/common/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/*:*/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/*:*/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/**:/usr/lib/jvm/java-7-openjdk-amd64/lib/:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar:HADOOP_CLASSPATH
>
>         simo@simo:~/hadoop-2.6.1$ <mailto:simo@simo:%7E/hadoop-2.6.1$>
>         bin/yarn classpath
>         /home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/conf:/home/simo/hadoop-2.6.1/share/hadoop/common/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/common/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/hdfs/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/lib/*:/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/*:/*home/simo/hadoop-2.6.1/share/hadoop/tools/lib/*:*/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/**:/usr/lib/jvm/java-7-openjdk-amd64/lib/:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar:HADOOP_CLASSPATH:/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*
>
>
>     Given that
>
>         simo@simo:~*/hadoop-2.6.1/share/hadoop/tools/lib*$ ls
>         hadoop-extras-2.6.1.jar servlet-api-2.5.jar
>         *hadoop-gridmix-2.6.1.jar* snappy-java-1.0.4.1.jar
>         hadoop-openstack-2.6.1.jar stax-api-1.0-2.jar
>         hadoop-rumen-2.6.1.jar                 xmlenc-0.52.jar
>         hadoop-sls-2.6.1.jar                   xz-1.0.jar
>         hadoop-streaming-2.6.1.jar zookeeper-3.4.6.jar
>
>     Am I doing something wrong ? How do I check the classpath of the
>     NodeManager ?
>
>     Greets
>
>     Simone
>
>     On 15/12/2015 12:45, Wellington Chevreuil wrote:
>>     Hi Simone,
>>
>>     You should make sure to include hadoop-rumen-2.6.0.jar on the
>>     classpath for the Nodemanagers, or include it on the classpath of
>>     your job.
>>
>>>     On 14 Dec 2015, at 09:56, siscia <sisciamirror@yahoo.com.INVALID
>>>     <mailto:sisciamirror@yahoo.com.INVALID>> wrote:
>>>
>>>     Hello folks,
>>>
>>>     I am trying to run a simulation with GridMix but the simulation
>>>     fails every time.
>>>
>>>     If I inspect the log I find the error with looks like this:
>>>
>>>         "Error:
>>>         *java.lang.ClassNotFoundException:
>>>         org.apache.hadoop.tools.rumen.ResourceUsageMetrics\n\tat*
>>>
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:366)\n\tat
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:355)\n\tat
>>>         java.security.AccessController.doPrivileged(Native
>>>         Method)\n\tat
>>>         java.net.URLClassLoader.findClass(URLClassLoader.java:354)\n\tat
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:425)\n\tat
>>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)\n\tat
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:358)\n\tat
>>>         java.lang.Class.getDeclaredConstructors0(Native
>>>         Method)\n\tat
>>>         java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)\n\tat
>>>         java.lang.Class.getConstructor0(Class.java:2895)\n\tat
>>>         java.lang.Class.getDeclaredConstructor(Class.java:2066)\n\tat org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)\n\tat
>>>         org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:66)\n\tat
>>>         org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)\n\tat
>>>         org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)\n\tat
>>>         org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)\n\tat
>>>         org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\n\tat
>>>         org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)\n\tat
>>>         java.security.AccessController.doPrivileged(Native
>>>         Method)\n\tat
>>>         javax.security.auth.Subject.doAs(Subject.java:415)\n\tat
>>>         org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)\n\tat
>>>         org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\n",
>>>
>>>     The error is pretty clear, but I have no idea how to fix it.
>>>
>>>     I run GridMix by command line with:
>>>
>>>         bin/hadoop jar
>>>         share/hadoop/tools/lib/hadoop-gridmix-2.6.1.jar iopath
>>>         trace.json
>>>
>>>     Any hits ?
>>>
>>>     Cheers :)
>>>
>>>     Simone
>>
>


Mime
View raw message