spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-6187) Report full executor exceptions to the driver
Date Thu, 05 Mar 2015 16:53:38 GMT

     [ https://issues.apache.org/jira/browse/SPARK-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-6187:
-----------------------------
    Priority: Minor  (was: Major)

It already does this:

{{sc.parallelize(Array(1,2,3)).map { i => throw new IllegalStateException() }.collect}}

results in ..

{code}
15/03/05 08:51:15 INFO DAGScheduler: Job 0 failed: collect at <console>:13, took 0.960206
s
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed
4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, xxx.cloudera.com): java.lang.IllegalStateException
	at $iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:13)
	at $iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:13)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:11
{code}

It looks like this failure happened so early, like when the task was even instantiated, that
it wasn't able to report back the error. I don't know if it's easy or hard to handle that
case too.

> Report full executor exceptions to the driver
> ---------------------------------------------
>
>                 Key: SPARK-6187
>                 URL: https://issues.apache.org/jira/browse/SPARK-6187
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.2.1
>            Reporter: Piotr Kołaczkowski
>            Priority: Minor
>
> If the task fails for some reason, the driver seems to report only the top-level exception,
without the cause(s). While it is possible to recover the full stacktrace from executor's
logs, it is quite annoying and would be better to just report the full stacktrace, with all
the causes to the driver application.
> Example stacktrace I just got reported by the application:
> {noformat}
>    org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage
0.0 failed 1 times, most recent failure: Lost task 5.0 in stage 0.0 (TID 5, localhost): java.lang.NoClassDefFoundError:
Could not initialize class org.apache.cassandra.db.Keyspace
>     	at com.datastax.bdp.spark.writer.BulkTableWriter.writeSSTables(BulkTableWriter.scala:194)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter.write(BulkTableWriter.scala:223)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:280)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:280)
>     	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>     	at org.apache.spark.scheduler.Task.run(Task.scala:56)
>     	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
>     	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> As you see, this is not very informative.
> In fact, the real exception is:
> {noformat}
> java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.db.Keyspace
>     	at com.datastax.bdp.spark.writer.BulkTableWriter.writeSSTables(BulkTableWriter.scala:194)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter.write(BulkTableWriter.scala:227)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284)
>     	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>     	at org.apache.spark.scheduler.Task.run(Task.scala:56)
>     	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
>     	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     	at java.lang.Thread.run(Thread.java:745)
>     java.lang.ExceptionInInitializerError
>     	at com.datastax.bdp.spark.writer.BulkTableWriter.writeSSTables(BulkTableWriter.scala:194)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter.write(BulkTableWriter.scala:227)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284)
>     	at com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284)
>     	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>     	at org.apache.spark.scheduler.Task.run(Task.scala:56)
>     	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
>     	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     	at java.lang.Thread.run(Thread.java:745)
>     Caused by: java.lang.NullPointerException
>     	at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:741)
>     	at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:72)
>     	... 10 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message