hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3296) Some levels are skipped while creating the task cache in JobInProgress
Date Thu, 08 May 2008 07:40:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595138#action_12595138

Amar Kamat commented on HADOOP-3296:

bq. Minimally, calling the total number of maps scheduled "Local map tasks" is confusing
Note that the counter will be incremented only if there is a locality at any level. Its different
from calling all the maps as LOCAL. I agree that we should change the counter name. The intention
is to show how many tasks got picked up from the cache. So it includes node-local, rack-local,
switch-local etc. I think _Aggregate Locality_ sounds better, comments?
bq. For which users would this be valuable information?
In general when the users have cache topology more than 2 levels.
bq. If there are more than two levels in the task cache and the distinction is significant,
how is an aggregate counter resolving the ambiguity?
Consider a case where there is no locality at the node level and also at the rack level. As
per the trunk there is no way to infer whether the scheduling went correctly or not. With
this aggregate counter one can check if the maps were from the cache or not. So the counter
is just a count of how may maps got picked up from the task cache.

> Some levels are skipped while creating the task cache in JobInProgress
> ----------------------------------------------------------------------
>                 Key: HADOOP-3296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3296
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>         Attachments: HADOOP-3296-v2.patch, HADOOP-3296.patch
> Consider the following piece of code
> {code:title=JobInProgress.createCache()|borderStyle=solid}
> Node node = jobtracker.resolveAndAddToTopology(host);
> for (int j = 0; j < maxLevel; j++) {
>           node = JobTracker.getParentNode(node, j);
>           .....
> {code}
> With {{maxLevel > 2}} the caches will be created in the following order
> ||j||node-level||
> |0|0|
> |1|1|
> |2|3|
> |3|6|
> which is not as desired.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message