hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6953) Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources()
Date Wed, 30 Aug 2017 17:04:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147614#comment-16147614
] 

Daniel Templeton commented on YARN-6953:
----------------------------------------

Getting better.  Still seems rather complex.  To add to the confusion, here's what I would
do:

{code}
  private static void setAllocationForMandatoryResources(
      Map<String, ResourceInformation> res, Configuration conf) {
    ResourceInformation mem = res.get(ResourceInformation.MEMORY_MB.getName());

    if (mem == null) {
      //complain
    } else {
      mem.setMinimumAllocation(getAllocation(conf,
          YarnConfiguration.RESOURCE_TYPES + "." + mem.getName() + MINIMUM_ALLOCATION,
          YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
          YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB));
      mem.setMaximumAllocation(getAllocation(conf,
          YarnConfiguration.RESOURCE_TYPES + "." + mem.getName() + MAXIMUM_ALLOCATION,
          YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB,
          YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB));
    }

    ResourceInformation cpu = res.get(ResourceInformation.MEMORY_MB.getName());

    if (cpu == null) {
      //complain
    } else {
      cpu.setMinimumAllocation(getAllocation(conf,
          YarnConfiguration.RESOURCE_TYPES + "." + cpu.getName() + MINIMUM_ALLOCATION,
          YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_VCORES,
          YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_VCORES));
      cpu.setMaximumAllocation(getAllocation(conf,
          YarnConfiguration.RESOURCE_TYPES + "." + cpu.getName() + MAXIMUM_ALLOCATION,
          YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES,
          YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES));
    }
  }

  private static void getAllocation(Configuration conf, String resourceTypesKey,
      String schedulerKey, long schedulerDefault) {
    long value = conf.getLong(resourceTypesKey, -1L);

    if (value == -1) {
      value = conf.getLong(schedulerKey, schedulerDefault);
    }

    return value;
  }
{code}

I didn't actually look to see if the null checks are needed.

This code is a little longer, but for me it's clearer.  I don't have to do any mental gymnastics
to understand what it's doing.

> Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources()
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6953
>                 URL: https://issues.apache.org/jira/browse/YARN-6953
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: YARN-3926
>            Reporter: Daniel Templeton
>            Assignee: Manikandan R
>            Priority: Minor
>              Labels: newbie
>         Attachments: YARN-6953-YARN-3926.001.patch, YARN-6953-YARN-3926.002.patch, YARN-6953-YARN-3926.003.patch,
YARN-6953-YARN-3926.004.patch, YARN-6953-YARN-3926-WIP.patch
>
>
> The {{setMinimumAllocationForMandatoryResources()}} and {{setMaximumAllocationForMandatoryResources()}}
methods are quite convoluted.  They'd be much simpler if they just handled CPU and memory
manually instead of trying to be clever about doing it in a loop.  There are also issues,
such as the log warning always talking about memory or the last element of the inner array
being a copy of the first element.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message