hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
Date Fri, 03 Apr 2015 13:44:54 GMT

    [ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394437#comment-14394437

Junping Du commented on YARN-3437:

Thanks [~sjlee0] for delivering a patch here! 
Just quickly go through the patch, looks like we are generating one app collector per map
task. I think this is good for scalability test on backend storage which can be a bottleneck
in mainstream cases. In addition, do we want to address some extreme cases, e.g. a huge applications
will have hundreds of thousands or even millions tasks? If so, then may be we want to know
a single app collector's bottleneck as well for accepting/forwarding messages from hundreds
of thousands maps. Also, in a real cluster, the mapping from cluster to app, and app to tasks
are all 1-N mapping. May be making app aggregator number configurable (just like map task
number, and byte per map, etc.) is something we can do for next step?
BTW, it has some duplicated code with YARN-2556 (like TimelineServerPerformance.java). Looks
like YARN-2556 is in pretty good shape and possible to go to trunk and branch-2 quickly. I
would remind to keep watching that JIRA status and do necessary rebase work if that patch
go in and we may want to merge it into YARN-2928 branch soon.

> convert load test driver to timeline service v.2
> ------------------------------------------------
>                 Key: YARN-3437
>                 URL: https://issues.apache.org/jira/browse/YARN-3437
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: YARN-3437.001.patch
> This subtask covers the work for converting the proposed patch for the load test driver
(YARN-2556) to work with the timeline service v.2.

This message was sent by Atlassian JIRA

View raw message