hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2672) Improve Gridmix (synthetic generator + reservation support)
Date Fri, 10 Oct 2014 00:30:33 GMT

    [ https://issues.apache.org/jira/browse/YARN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166055#comment-14166055
] 

Carlo Curino commented on YARN-2672:
------------------------------------

Quick how to use:

I usually run it with something like this:

{code:title=gridmix.sh|borderStyle=solid}
#!/bin/bash

TRACE=${1:-syn.json}
LOCATION=${2:-/user/hadoop/gridmix100g}
. env.sh
date
hadoop fs -rm -r $LOCATION/gridmix
hadoop fs -rm /user/hadoop/$TRACE
hadoop fs -put $TRACE /user/hadoop
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_YARN_HOME/share/hadoop/tools/lib/*:/home/hadoop/commons-math3-3.0.jar

echo "launching gridmix"

hadoop jar $HADOOP_COMMON_HOME/share/hadoop/tools/lib/hadoop-gridmix-3.0.0-SNAPSHOT.jar -libjars
$HADOOP_COMMON_HOME/share/hadoop/tools/lib/hadoop-rumen-3.0.0-SNAPSHOT.jar \
  -Dgridmix.job-producer.is.synthetic=true \
  -Dgridmix.job-submission.policy=REPLAY \
  -Dgridmix.job.type=LOADJOB \
  -Dgridmix.job-submission.default-queue=default \
  -Dgridmix.sleep.fake-locations=3 \
  -Dgridmix.compression-emulation.enable=false \
  -Dgridmix.job.seq=1 \
  -Dgridmix.client.submit.threads=20 \
  -Dgridmix.client.pending.queue.depth=10 \
  -Dmapreduce.map.java.opts=-Xmx2000m \
  -Dmapreduce.reduce.java.opts=-Xmx4000m \
   $LOCATION /user/hadoop/$TRACE

{code}

The syn.json looks something like this:

{code:title=syn.json|borderStyle=solid}
{
  "description" : "tiny jobs workload",
  "num_jobs" : 1000,
  "rand_seed" : 2,
  "workloads" : [
    {
      "workload_name" : "tiny-test",
      "workload_weight": 0.5,
      "description" : "Sort jobs",
      "queue_name" : "dedicated",
      "job_classes" : [
        {
          "class_name" : "class_1",
          "class_weight" : 1.0,

          "mtasks_avg" : 5,
          "mtasks_stddev" : 1,
          "rtasks_avg" : 5,
          "rtasks_stddev" : 1,

          "in_avg" : 1048500,
          "in_stddev" : 17466,
          "shuffle_avg" : 104085000,
          "shuffle_stddev" : 162666,
          "out_avg" : 10485700,
          "out_stddev" : 1876000,
          "dur_avg" : 600,
          "dur_stddev" : 60,

          "mtime_avg" : 3,
          "mtime_stddev" : 60,
          "rtime_avg" : 3,
          "rtime_stddev" : 6,

          "map_max_memory_avg" : 1024,
          "map_max_memory_stddev" : 0.001,
          "reduce_max_memory_avg" : 1024,
          "reduce_max_memory_stddev" : 0.001,
          "bytes_per_map_record" : 512,
          "bytes_per_shuffle_record" : 512,
          "bytes_per_reduce_record" : 1024,
          
          "chance_of_reservation" : 1.0,
          "deadline_factor_avg" : 10.0,
          "deadline_factor_stddev" : 0.001,
          "gang_size" : 1
        }
       ],
      "time_distribution" : [
        { "time" : 1, "jobs" : 100 },
        { "time" : 3600, "jobs" : 0 }
     ]
        }
 ]  
}   
{code}

The distributions of each parameter are *Normal* with average X_avg and standard deviation
X_stddev.

> Improve Gridmix (synthetic generator + reservation support)
> -----------------------------------------------------------
>
>                 Key: YARN-2672
>                 URL: https://issues.apache.org/jira/browse/YARN-2672
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler, fairscheduler, resourcemanager
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-2672.patch
>
>
> This JIRA proposes an enhancement of Gridmix that contains:
> 1) a synthetic generator to produce load without the need of a trace, but based on distributions
> 2) include negotiation of reservations (to test YARN-1051). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message