spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrei Taleanu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-20321) Spark UI cannot be shutdown in spark streaming app
Date Thu, 13 Apr 2017 11:27:41 GMT
Andrei Taleanu created SPARK-20321:
--------------------------------------

             Summary: Spark UI cannot be shutdown in spark streaming app
                 Key: SPARK-20321
                 URL: https://issues.apache.org/jira/browse/SPARK-20321
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.1.0
            Reporter: Andrei Taleanu


When an exception thrown the transform stage is handled in foreachRDD and the streaming context
is forced to stop, the SparkUI appears to hang and continually dump the following logs in
an infinite loop:
{noformat}
...
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager
- Selector loop woken up from select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager
- Selector loop waiting on select
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager
- Selector loop woken up from select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager
- Selector loop waiting on select
...
{noformat}

Unfortunately I don't have a minimal example that reproduces this issue but here is what I
can share:
{noformat}
val dstream = pull data from kafka
val mapped = dstream transform { rdd =>
  val data = getData // Perform a call that potentially throws an exception
  // broadcast the data
  // flatMap the RDD using the data
}

mapped.foreachRDD {
  try {
    // write some data in a DB
  } catch {
    case t: Throwable =>
      dstream.context.stop(stopSparkContext = true, stopGracefully = false)
  }
}

mapped.foreachRDD {
  try {
    // write data to Kafka
    // manually checkpoint the Kafka offsets (because I need them in JSON format)
  } catch {
    case t: Throwable =>
      dstream.context.stop(stopSparkContext = true, stopGracefully = false)
  }
}
{noformat}

The issue appears when stop is invoked. At the point when SparkUI is stopped, it enters that
infinite loop. Initially I thought it relates to Jetty, as the version used in SparkUI had
some bugs (e.g. [this one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped
Jetty to a more recent version (March 2017) but still got the error.

I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message