hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3296) Some levels are skipped while creating the task cache in JobInProgress
Date Wed, 07 May 2008 21:01:57 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595040#action_12595040
] 

Chris Douglas commented on HADOOP-3296:
---------------------------------------

bq. The reason is that if there are more than 2 levels in the task cache (e.g local/rack/gateway/...)
then there is no way to know if the maps were scheduled locally (at a level above rack) or
not.

Minimally, calling the total number of maps scheduled "Local map tasks" is confusing. I'll
grant without hesitation that hard-coding two levels doesn't cover all topologies, but this
counter provides only slightly more information than the existing counters, diluted. For which
users would this be valuable information? If there are more than two levels in the task cache
and the distinction is significant, how is an aggregate counter resolving the ambiguity? Please
correct this impression if it's mistaken, but this counter is a partial fix to an issue orthogonal
to the one described in this JIRA; it should be left out.

> Some levels are skipped while creating the task cache in JobInProgress
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-3296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3296
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>         Attachments: HADOOP-3296-v2.patch, HADOOP-3296.patch
>
>
> Consider the following piece of code
> {code:title=JobInProgress.createCache()|borderStyle=solid}
> Node node = jobtracker.resolveAndAddToTopology(host);
> for (int j = 0; j < maxLevel; j++) {
>           node = JobTracker.getParentNode(node, j);
>           .....
> {code}
> With {{maxLevel > 2}} the caches will be created in the following order
> ||j||node-level||
> |0|0|
> |1|1|
> |2|3|
> |3|6|
> which is not as desired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message