hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
Date Fri, 14 Apr 2017 16:39:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969231#comment-15969231
] 

Carlo Curino commented on YARN-6363:
------------------------------------

I have fixed and tested the commandline verifying both new and old options, and nodes/no-nodes
as follows: 

{code}
# MANUALLY prepare copying source and data-files (with something like):
# cp -r <path-to-source>/hadoop-dist/target/hadoop-${COMMON_VERSION} $BASEDIR
# cp <path-to-source>/hadoop-tools/hadoop-sls/src/test/resources/*.json $BASEDIR/data

BASEDIR=/home/ccurino/hadoop-deployed
COMMON_VERSION=3.0.0-alpha3-SNAPSHOT

export HADOOP_CONF_DIR=$BASEDIR/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/sample-conf/
export YARN_CONF_DIR=$BASEDIR/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/sample-conf/
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:${BASEDIR}/hadoop-${COMMON_VERSION}/bin:${BASEDIR}/hadoop-${COMMON_VERSION}/sbin:${BASEDIR}/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/bin

# NO NODES OLD OPTIONS
timeout -s 9 60s slsrun.sh --input-rumen=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --input-sls=$BASEDIR/data/inputsls.json --output-dir=/tmp/sls-out2

# NO NODES NEW OPTIONS
timeout -s 9 60s slsrun.sh --tracetype=RUMEN --tracelocation=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SLS --tracelocation=$BASEDIR/data/inputsls.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SYNTH --tracelocation=$BASEDIR/data/syn.json --output-dir=/tmp/sls-out2

# WITH NODES OLD OPTIONS
timeout -s 9 60s slsrun.sh --input-rumen=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --input-sls=$BASEDIR/data/inputsls.json --nodes=$BASEDIR/data/nodes.json
--output-dir=/tmp/sls-out2

# WITH NODES NEW OPTIONS
timeout -s 9 60s slsrun.sh --tracetype=RUMEN --tracelocation=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SLS --tracelocation=$BASEDIR/data/inputsls.json --nodes=$BASEDIR/data/nodes.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SYNTH --tracelocation=$BASEDIR/data/syn.json --nodes=$BASEDIR/data/nodes.json
--output-dir=/tmp/sls-out2 
{code}

I ran and checked all the configurations below, using default configs from share/tools/sls/sample-conf,
and the data from sample-data and src/test/resources/*.json.
Manually inspecting the runs things look good, RM comes up with Webapp, nodes are loaded as
expected, and apps get to run for all but RUMEN configurations (more below).
The new TestSLSRunner also introduces more checks (with both schedulers). In future JIRAs
we can further extend this to leverage YARN-6451 to have a deeper introspection/check of these
runs. 

The RUMEN version seems to have a problem parsing the rumen example trace (nodes are loaded
only from --nodes, and jobs are never loaded). 
Turns out this problem is also present in trunk (running the TestSLSRunner does bring the
RM up, but doesn't run any job nor loads nodes)---I am opening a separate JIRA to track this
existing bug YARN-6482.

(Per [~subru] offline comment, I am converting the attached overview document in additions
to the SLS documentation, will upload soon)


> Extending SLS: Synthetic Load Generator
> ---------------------------------------
>
>                 Key: YARN-6363
>                 URL: https://issues.apache.org/jira/browse/YARN-6363
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, YARN-6363.v10.patch,
YARN-6363.v11.patch, YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch, YARN-6363.v4.patch,
YARN-6363.v5.patch, YARN-6363.v6.patch, YARN-6363.v7.patch, YARN-6363.v9.patch
>
>
> This JIRA tracks the introduction of a synthetic load generator in the SLS. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message