spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-11296) FutureAction.onComplete consumes a thread while waiting for job
Date Sat, 24 Oct 2015 19:04:27 GMT


Apache Spark commented on SPARK-11296:

User 'reggert' has created a pull request for this issue:

> FutureAction.onComplete consumes a thread while waiting for job
> ---------------------------------------------------------------
>                 Key: SPARK-11296
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Scheduler, Spark Core
>    Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1
>            Reporter: Richard W. Eggert II
>              Labels: performance
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> The implementations of the onComplete callback (and, by extension, the callbacks based
on it) of FutureAction (SimpleFutureAction and ComplexFutureAction) each consume a thread
that blocks waiting for the underlying job(s) to complete. This is unnecessary and is likely
to be very surprising to anyone accustomed to the usual behavior of Scala Futures (i.e., that
callbacks only occupy threads AFTER the underlying task has completed),  and it runs the risk
of causing thread starvation for any processes utilizing the same thread pool (especially
if is used). The implementations can and should be
reworked to operate in a non-blocking manner, by modifying the underlying JobWaiter class
to expose a non-blocking API, so that the onComplete (as well as onSuccess and onFailure)
callbacks only utilize threads while they are actually executing.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message