spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sryza <>
Subject [GitHub] spark pull request: [SPARK-6954] [YARN] Dynamic allocation: numExe...
Date Tue, 21 Apr 2015 01:58:14 GMT
Github user sryza commented on the pull request:
    I've thought about this more and think I'm able to articulate why I think the present
fix is incorrect.
    Aside from the weirdness of the idea that there can be a negative number of pending executors,
I think the behavior that you (Andrew) describe in your example is not a problem.  numPendingExecutors
ends up as -10.  targetNumExecutors is then calculated as numPendingExecutors + executorIds.size
- executorsPendingRemoval.size, which comes out to 0.  This is the correct number that we'd
like to pass on to YarnAllocator, which tells YarnAllocator that if executors die naturally
or are preempted, it shouldn't try to replace them because there's no need.  If we simply
force numExecutorsPending to be 0 in that situation, targetNumExecutors would be calculated
as 10, which is incorrect.
    The problem is that some sort of double counting is going on.  I think executors that
we don't want can be counted both in the form of negative numPendingExecutors and as part
of executorsPendingRemoval.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message