spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JoshRosen <>
Subject [GitHub] spark pull request: [SPARK-3626] [WIP] Replace AsyncRDDActions wit...
Date Sun, 21 Sep 2014 22:23:23 GMT
Github user JoshRosen commented on the pull request:
    I've taken another pass at this.  This time, I kept AsyncRDDActions but re-implemented
it using `runAsync`, but I'm actually on the fence about that change.  The one difference
here is that the asynchronous jobs will now be submitted with anonymous job groups rather
than as part of the calling thread's job group.  This change might be observable by a user
who writes a job that fires off multiple asynchronous actions from a single driver control
thread, then attempts to cancel that thread's job group.  Because job groups don't have any
hierarchy / nesting, this would break the cancellation of those jobs.
    I'm beginning to get the sense that we might not have much room to change anything about
the implementation of AsyncRDDActions, so maybe we should just let them be.
    @rxin Based on our discussion, I added a check in DAGScheduler to reject jobs submitted
by cancelled threads.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message