hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
Date Thu, 03 Sep 2015 14:04:46 GMT

     [ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Varun Vasudev updated YARN-4081:
    Attachment: YARN-4081-YARN-3926.003.patch

1) Uses String instead of URI for resource key? I think it maybe more efficient to use String,
it will be easier to construct when using it, uses less resource (hasn't tested, but I think
it will be true according to #fields in String and URI). I can understand the motivation of
solving conflicts of resource namespace, but I think namespace conflict is not the major use
case AND String can define namespace as well.


2) Relationship between ResourceInformation and ResourceMapEntry: currently it's 1-1 mapping,
a ResourceInformation has value/unit from ResourceMapEntry, they're kind of overlapping and
also confusing. I think it's better to make ResourceInformation to be one for each resource
type. ResourceMapEntry contains runtime information, and ResourceInformation contains configured
information. This will also avoid create ResourceInformation instance when invoking Resource.getResourceInformation()

3) Resource unit: I like the design which can easily convert a internal value to human-readable
value. But I think maybe we don't need to support define unit in ResourceMapEntry. There're
some cons of it:

    When we doing comparision of resources, we have to convert units, it's an extra overhead.
    It doesn't make a lot of sense to me that keep internal unit of resources: We should handle
it when constructing Resource (something like Resource.newInstance("memory", 12, "GB")). And
we will use the standard unit to do internal computations.
    We can define the standard unit in each "ResourceInformation" if you agree with #2.

I spoke with Wangda offline an we agree that it makes more sense to do performance testing
once we have the DRC changes in. Since this patch is going in to a branch, there's no issue
committing and running a full suite of performance tests once we have the DRC changes in.

4) Do you think it's better to have a global ResourceInformation map instead of storing it
in each Resource instance?

For now, I'd like to keep it per-resource instance but if it becomes an overhead, we can make
it a global instance.

5) Resource#compareTo/hashCode has debug logging.


6) It seems not necessary to instance ArrayList in Resource#compareTo. Just traverse the set
can avoid create the temporary ArrayList.
Good point. I decided to use the size of the set itself as the sort order and avoid the issue

> Add support for multiple resource types in the Resource class
> -------------------------------------------------------------
>                 Key: YARN-4081
>                 URL: https://issues.apache.org/jira/browse/YARN-4081
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: YARN-4081-YARN-3926.001.patch, YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch
> For adding support for multiple resource types, we need to add support for this in the
Resource class.

This message was sent by Atlassian JIRA

View raw message