flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xintong Song (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-19556) YarnResourceManagerDriver.getPendingRequestsAndCheckConsistency only returns container requests for first matching equivalent resource
Date Sat, 10 Oct 2020 03:26:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211527#comment-17211527
] 

Xintong Song commented on FLINK-19556:
--------------------------------------

I think [~karmagyz] is correct.

The current implementation cannot support containers with different resources, because (as
described in FLINK-19324) Hadoop 2.8 and previous versions do not support requesting containers
of different resources with the same priority. To support containers with different resources,
we need to request them with different priorities anyway. Therefore, using priority for identifying
containers with different resources would be much simpler than the current resource matching,
and we won't have problem with \{{AMRMClientAsync.getMatchingRequests}} because the priority
is strictly matched.

The only concern I see in using priority for mapping requested/allocated containers is that,
we might enforce some sort of prioritization over the containers. IMO, this is a necessary
price for supporting containers with different resources on Hadoop 2.8 and below. We can discuss
this in FLINK-19324.

WDYT, [~trohrmann]?

> YarnResourceManagerDriver.getPendingRequestsAndCheckConsistency only returns container
requests for first matching equivalent resource
> --------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-19556
>                 URL: https://issues.apache.org/jira/browse/FLINK-19556
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / YARN
>    Affects Versions: 1.12.0
>            Reporter: Till Rohrmann
>            Priority: Major
>             Fix For: 1.12.0
>
>
> The {{YarnResourceManagerDriver}} will retrieve the number of pending container requests
for a given {{Resource}} by calling {{YarnResourceManagerDriver.getPendingRequestsAndCheckConsistency(Resource)}}.
Part of this method is to find the equivalent resources for the given resource. Next we find
for every equivalent resource the matching container requests. Instead of unifying these collection
of container requests, the method returns the first non-empty collection. This might return
an incomplete set of container requests if the Yarn respects the vcores of resources given
to {{AMRMClientAsync.getMatchingRequests}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message