hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2014) Performance: AM scaleability is 10% slower in 2.4 compared to 0.23.9
Date Tue, 13 May 2014 15:23:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996494#comment-13996494

Jason Lowe commented on YARN-2014:

I did a bit of investigation on this, and the problem appears to be around the duration of
the tasks.  In 2.4 the sleep job tasks are taking about 660 msec longer to execute than they
do in 0.23.  I didn't nail down exactly where this extra delay was coming from, but I did
notice that the tasks in 2.4 are loading over 800 more classes than they do in 0.23.  I think
most of these are coming from the service loader for FileSystem schemas, as the 2.4 tasks
loads every FileSystem available and 0.23 does not.  In 0.23 FileSystem schemas are declared
in configs, but in 2.4 they are dynamically detected and loaded via a service loader.

The ~0.5s delay in the task appears to be a fixed startup cost and is amplified by the AM
scalability test since it runs very short tasks (the main portion of the map task lasts 1
second) and multiple tasks are run per map "slot" on the cluster, serializing the task startup

> Performance: AM scaleability is 10% slower in 2.4 compared to 0.23.9
> --------------------------------------------------------------------
>                 Key: YARN-2014
>                 URL: https://issues.apache.org/jira/browse/YARN-2014
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: patrick white
> Performance comparison benchmarks from 2.x against 0.23 shows AM scalability benchmark's
runtime is approximately 10% slower in 2.4.0. The trend is consistent across later releases
in both lines, latest release numbers are:
>     runtime 255.6 seconds (avg 5 passes)
> runtime 230.4 seconds (avg 5 passes)
> Diff: -9.9% 
> AM Scalability test is essentially a sleep job that measures time to launch and complete
a large number of mappers.
> The diff is consistent and has been reproduced in both a larger (350 node, 100,000 mappers)
perf environment, as well as a small (10 node, 2,900 mappers) demo cluster.

This message was sent by Atlassian JIRA

View raw message