spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neelesh Srinivas Salian (JIRA)" <>
Subject [jira] [Commented] (SPARK-12265) Spark calls System.exit inside driver instead of throwing exception
Date Thu, 10 Dec 2015 16:31:11 GMT


Neelesh Srinivas Salian commented on SPARK-12265:

[~srowen] [~dragos]
Can `SparkException` be a good wrapper for all such cases? 

Also, I know Yarn has YarnRuntimeException which is a wrapper in all cases that throw Runtime.
Would it be a good suggestion to have such an implementation in Spark (unless already present)?

> Spark calls System.exit inside driver instead of throwing exception
> -------------------------------------------------------------------
>                 Key: SPARK-12265
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Mesos
>    Affects Versions: 1.6.0
>            Reporter: Iulian Dragos
> Spark may call {{System.exit}} if Mesos sends an error code back to the MesosSchedulerDriver.
This makes Spark very hard to test, since this effectively kills the driver application under
test. Such tests may run under ScalaTest, that doesn't get a chance to collect a result and
populate a report.
> Relevant code is in MesosSchedulerUtils.scala:
> {code}
>             val ret =
>             logInfo(" returned with code " + ret)
>             if (ret != null && ret.equals(Status.DRIVER_ABORTED)) {
>               System.exit(1)
>             }
> {code}
> Errors should be signaled with a {{SparkException}} in the correct thread.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message