hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guo Leitao <leitao....@gmail.com>
Subject Re: Gridmix on Yarn
Date Fri, 01 Aug 2014 13:16:06 GMT
Hi Brian, I'm using hadoop-2.3.0-cdh5.1.0, there is gridmix jar in this
package.

step 1: using Rumen to generate the job trace file.

sudo -u yarn java -cp `hadoop classpath`
org.apache.hadoop.tools.rumen.TraceBuilder
file:///tmp/jobhistory_log/job-trace.json
file:///tmp/jobhistory_log/topology.output
file:///tmp/jobhistory_log/15/000375/

step 2: run gridmix on yarn+mrv2

sudo -u yarn hadoop dfs -put /tmp/jobhistory_log/job-trace.json /tmp
sudo -u yarn hadoop jar /usr/lib/hadoop-mapreduce/hadoop-gridmix.jar
-Dgridmix.min.file.size=10485760
-Dgridmix.job-submission.use-queue-in-trace=true
-Dgridmix.distributed-cache-emulation.enable=false  -generate 133120m
hdfs:///user/yarn/foo/ hdfs:///tmp/job-trace.json

hope this can be helpful.

-Leitao

2014-08-01 20:08 GMT+08:00 Brian Husted <brian.husted@gmail.com>:

> Hi Guo,
>
> Can you please let me know if there was any specific configuration needed
> to get Gridmix working with YARN+MRv2.   We are getting the following
> exception:
>
> INFO gridmix.JobSubmitter: Job org.apache.hadoop.mapreduce.Job@18a8ce2
> submission failed java.lang.ArithmeticException: / by zero
> at
>
> org.apache.hadoop.mapred.gridmix.GenerateData$GenDataFormat.getSplits(GenerateData.java:161)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:902)
>
> It seems to be very similar to this issue which relates to not having the
> tasktracker is not running
>
> https://issues.apache.org/jira/browse/MAPREDUCE-2016
>
>
>
>
>
> --------------------------------------------------------
> Yes, I'm using Gridmix running on YARN+MRv2.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message