hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi" <gr...@yahoo-inc.com>
Subject RE: Generating job and topology traces from history folder of multinode cluster using Rumen
Date Fri, 16 Dec 2011 06:57:25 GMT
Amar is working on this issue MAPREDUCE-3349. The patch is not comiited to trunk yet. Feel
free to try it out while it gets reviewed and committed.

-Ravi
________________________________________
From: arun k [arunk786@gmail.com]
Sent: Friday, December 16, 2011 12:22 PM
To: mapreduce-user@hadoop.apache.org
Subject: Generating job and topology traces from history folder of multinode cluster using
Rumen

Hai guys !

I have set up 5 node cluster with each of them in different racks.

I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder using
Main Class: org.apache.hadoop.tools.rumen.TraceBuilder
I ran some jobs on cluster and  used copy of /usr/local/hadoop/logs/history folder of master
node as input to TraceBuilder.
Arguments: /home/arun/job.json  /home/arun/topology.json  /home/ubuntu/Documents/testlog

But i get
11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while processing the
[possibly virtual] file master_1324011575958_job_201112161029_0001_hduser_word+count within
Path file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count
java.lang.NullPointerException
        at org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
        at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149)
        at org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310)
        at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
        at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while processing the
[possibly virtual] file master_1324011575958_job_201112161029_0004_hduser_word+count within
Path file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count
java.lang.NullPointerException
        at org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
        at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde .........
         ....................

It generates job trace json file but the topology trace json file doesn't have 5 node's info
and is like this :

{
  "name" : "<root>",
  "children" : [ ]
}

> Which history folder do i need to use ? masters or slaves ?  how do i use it ?
> I have a single  node cluster from which i got logs and gave input TraceBuilder and it
works.
> How do i run tracebuilder from terminal and do i need to use only from version 21 ?

Can anyone help me out ?


Thanks,
Arun

Mime
View raw message