spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JoshRosen <>
Subject [GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...
Date Tue, 03 Feb 2015 03:26:45 GMT
Github user JoshRosen commented on the pull request:
    > If we have tasks at some high locality level that cannot be scheduled at their preferred
locality and there are tasks at a lower locality level that can be scheduled, then I guess
we might be concerned about whether scheduling the less-local tasks could rob tasks waiting
at a higher locality level of their opportunity to run. This shouldn't happen, though, since
those tasks will already have been offered those resources and turned them down.
    Actually, on reflection, I guess this gets trickier when you have multiple task sets:
in this case, you want to ensure that _all_ unscheduled process-local tasks have turned down
a resource before you allow _any_ node-local task to obtain it.
    I'm going to try to trace through the interaction between DAGScheduler and TaskSetManager
to try to understand how we handle this case.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message