hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
Date Fri, 01 Aug 2014 23:21:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083164#comment-14083164
] 

Jian Fang commented on YARN-2380:
---------------------------------

Seems DominantResourceCalculator sets vcore correctly. But the capacity scheduler in YARN
uses DefaultResourceCalculator for yarn.scheduler.capacity.resource-calculator by default.


Anyway, resetting vcore to 1 is an unexpected behavior, which could create a lot of troubles.
Either fix DefaultResourceCalculator or just remove it.


> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---------------------------------------------------------------------------
>
>                 Key: YARN-2380
>                 URL: https://issues.apache.org/jira/browse/YARN-2380
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Jian Fang
>            Priority: Critical
>
> I added some log info to the method normalizeRequest() as follows.
>   public static void normalizeRequest(
>       ResourceRequest ask, 
>       ResourceCalculator resourceCalculator, 
>       Resource clusterResource,
>       Resource minimumResource,
>       Resource maximumResource,
>       Resource incrementResource) {
>     LOG.info("Before request normalization, the ask capacity: " + ask.getCapability());
>     Resource normalized = 
>         Resources.normalize(
>             resourceCalculator, ask.getCapability(), minimumResource,
>             maximumResource, incrementResource);
>     LOG.info("After request normalization, the ask capacity: " + normalized);
>     ask.setCapability(normalized);
>   }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils
(IPC Server handler 4 on 9024): Before request normalization, the ask capacity: <memory:1536,
vCores:2>
> 2014-08-01 20:54:15,537 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils
(IPC Server handler 4 on 9024): After request normalization, the ask capacity: <memory:1536,
vCores:1>
> The root cause is the DefaultResourceCalculator calls Resources.createResource(normalizedMemory)
to regenerate a new resource with vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and the container
resource and many other potential issues if the user requests containers with vcore > 1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message