hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
Date Sat, 06 Jan 2018 19:52:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314876#comment-16314876

Wangda Tan commented on YARN-6599:

Thanks [~asuresh],

bq. Wangda Tan, apologize, but I think we really should split this part.

Let me explain a bit, it may not be as hard as you thought about. 

Following changes are related to partition:

1) Added nodePartition to PlacementConstraints 
2) Renamed acceptNodePartition to precheckNode so we can blacklist nodes in AppPlaementAllocator.
3) Check node partition inside new AppPlaementAllocator implementation.
4) Added partition check in PlacementConstraintsUtil.
5) Unit test related changes.

#1/#3 is very straightforward. #2 is needed anyway since we want to plug logics to check SchedulingRequest
while allocating/validating container allocation, regardless of splitting logic. To me, #3
is also not hard. Please let me know your thoughts on this. It may take me several hours to
split the logic, I'm not sure if it worths since we're pushing the branch merge soon.

bq. It just means source tag == target expression tag.
This looks not correct, first of all, it is not a new type, it's more like a syntactic sugar
which should be built inside PlacementConstraints, and I'm doubt if it is really needed. We
can discuss it in a separate JIRA.

bq. As I mentioned in the previous comment. There is is this issue of application priority
To me, this is not different from affinity/anti-affinity to different tags. For example a
Tensorflow "worker" affinity to "parameter server" which is in different priority. A wrong
priority setting could still cause deadlock issue. And in order to specify intra-app affinity/anti-affinity,
we have to include application id inside allocation tags since by default allocation tag query
should include all apps.

> Support rich placement constraints in scheduler
> -----------------------------------------------
>                 Key: YARN-6599
>                 URL: https://issues.apache.org/jira/browse/YARN-6599
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-6599-YARN-6592.003.patch, YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch,
YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, YARN-6599-YARN-6592.008.patch,
YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message