hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6522) Make SLS JSON input file format simple and scalable
Date Fri, 28 Apr 2017 19:18:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989320#comment-15989320
] 

Yufei Gu edited comment on YARN-6522 at 4/28/17 7:17 PM:
---------------------------------------------------------

Uploaded patch v1. It largely simplifies the job configuration. 
Here is a job configuration for FS minshare preemption test. What it looks like before patch:
{code}
{
    "am.type" : "mapreduce",
        "job.start.ms" : 0,
        "job.end.ms" : 60000,
        "job.queue.name" : "default",
        "job.id" : "job_1",
        "job.user" : "default",
        "job.tasks" : [ {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6664,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        } ]
}
{
    "am.type" : "mapreduce",
        "job.start.ms" : 0,
        "job.end.ms" : 60000,
        "job.queue.name" : "default",
        "job.id" : "job_11",
        "job.user" : "default",
        "job.tasks" : [ {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6664,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        } ]
}
{
    "am.type" : "mapreduce",
        "job.start.ms" : 10000,
        "job.end.ms" : 60000,
        "job.queue.name" : "min-wait",
        "job.id" : "job_2",
        "job.user" : "default",
        "job.tasks" : [ {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 10000,
            "container.end.ms" : 40000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 10000,
            "container.end.ms" : 40000,
            "container.priority" : 20,
            "container.type" : "map"
        } ]
}
{code}
and after patch:
{code}
{ "num_nodes":3}
{ "job.start.ms" : 0, "job.queue.name": "default", "job.tasks": [ {"count":29, "duration.ms":
50000}]}
{ "job.start.ms" : 10000, "job.queue.name": "min-wait", "job.tasks":[{"count":2, "duration.ms":
30000}]}
{code}


was (Author: yufeigu):
Uploaded patch v1. It largely simplifies the job configuration. 
Here is a job configuration for minshare preemption test. What it looks like before patch:
{code}
{
    "am.type" : "mapreduce",
        "job.start.ms" : 0,
        "job.end.ms" : 60000,
        "job.queue.name" : "default",
        "job.id" : "job_1",
        "job.user" : "default",
        "job.tasks" : [ {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6664,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        } ]
}
{
    "am.type" : "mapreduce",
        "job.start.ms" : 0,
        "job.end.ms" : 60000,
        "job.queue.name" : "default",
        "job.id" : "job_11",
        "job.user" : "default",
        "job.tasks" : [ {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6664,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node3",
            "container.start.ms" : 6665,
            "container.end.ms" : 60000,
            "container.priority" : 20,
            "container.type" : "map"
        } ]
}
{
    "am.type" : "mapreduce",
        "job.start.ms" : 10000,
        "job.end.ms" : 60000,
        "job.queue.name" : "min-wait",
        "job.id" : "job_2",
        "job.user" : "default",
        "job.tasks" : [ {
            "container.host" : "/default-rack/node1",
            "container.start.ms" : 10000,
            "container.end.ms" : 40000,
            "container.priority" : 20,
            "container.type" : "map"
        }, {
            "container.host" : "/default-rack/node2",
            "container.start.ms" : 10000,
            "container.end.ms" : 40000,
            "container.priority" : 20,
            "container.type" : "map"
        } ]
}
{code}
and after patch:
{code}
{ "num_nodes":3}
{ "job.start.ms" : 0, "job.queue.name": "default", "job.tasks": [ {"count":29, "duration.ms":
50000}]}
{ "job.start.ms" : 10000, "job.queue.name": "min-wait", "job.tasks":[{"count":2, "duration.ms":
30000}]}
{code}

> Make SLS JSON input file format simple and scalable
> ---------------------------------------------------
>
>                 Key: YARN-6522
>                 URL: https://issues.apache.org/jira/browse/YARN-6522
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler-load-simulator
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Yufei Gu
>            Assignee: Yufei Gu
>         Attachments: YARN-6522.001.patch
>
>
> SLS input format is verbose, and it doesn't scale out. We can improve it in these ways:
> # We need to configure tasks one by one if there are more than one task in a job, which
means the job configuration usually includes lots of redundant items. Specify the number task
for a task configuration will solve this issue.
> # Container host is useful for locality testing. It is obnoxious to specify container
host for each task for tests unrelated to locality. We would like to make it optional.
> # For most tests, we don't care about job.id. Make it optional and generated automatically
by default.
> # job.finish.ms doesn't make sense, just remove it.
> # container type and container priority should be optional as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message