hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Roelofs (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-2424) MR-279: counters/UI/etc. for uber-AppMaster (in-cluster LocalJobRunner for MRv2)
Date Fri, 15 Apr 2011 02:26:05 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Greg Roelofs updated MAPREDUCE-2424:

    Attachment: MR-2424-MR-1220-yarn.v2.MR-279-hadoop-yarn.patch.txt

Same as v1 except with new "Uberized: true/false" line in the header block of the single-job
GUI page.  (Many thanks to Luke Lu for basically coding the GUI change verbally. :-) )

The multi-job RM page (i.e., marking each job entry as either uberized or not for a quick
overview) is likely to be much harder.  Unlike the original UberTask approach, here the decision
to uberize simply changes some internal state within MRAppMaster after it's already running,
so one would need some sort of protocol for an AM to pass either state info or maybe just
a descriptive string to the RM. (Note that the latter would have some security implications;
the string would need to be sanitized and perhaps truncated at 40 [Unicode] characters or
something to avoid DoS issues. On the other hand, it's probably hard to come up with a sufficiently
general state mechanism that could accommodate future non-MR AMs without going full Avro/PB

> MR-279: counters/UI/etc. for uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> --------------------------------------------------------------------------------
>                 Key: MAPREDUCE-2424
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2424
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>            Reporter: Greg Roelofs
>            Assignee: Greg Roelofs
>             Fix For: 0.23.0
>         Attachments: MR-2424-MR-1220-yarn.v1.MR-279-hadoop-yarn.patch.txt, MR-2424-MR-1220-yarn.v2.MR-279-hadoop-yarn.patch.txt
> Polish uber-AM (MAPREDUCE-2405).  Specifically:
> * uber-specific counters ("command-line UI")
> * GUI indicators
> ** RM all-containers level
> ** multi-job app level [if exists]
> ** single-job level
> * fix uber-decision ("is this a small job?"):
> ** memory criterion
> ** input-bytes criterion
> * disable speculation
> * isUber() method (somewhere) for unit tests to use
> * delete (most of) old UberTask code (MAPREDUCE-1220; came in with initial MR-279 branch)
> * implement non-RPC, local version of umbilical
> * AM restart (default 4 tries) on another node on any task-attempt failure
> * uber-specific metrics?
> * rename configurables? (still "ubertask"-based)

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message