spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-22264) History server will be unavailable if there is an event log file with large size
Date Thu, 12 Oct 2017 10:08:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-22264:
------------------------------------

    Assignee:     (was: Apache Spark)

> History server will be unavailable if there is an event log file with large size
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-22264
>                 URL: https://issues.apache.org/jira/browse/SPARK-22264
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy
>    Affects Versions: 2.1.0
>            Reporter: zhoukang
>         Attachments: not-found.png
>
>
> History server will be unavailable if there is an event log file with large size.
> Large size here means the replaying time is too long.
> *We can fix this to add a timeout for event log replaying.*
> *Here is an example:*
> Every application submitted after restart can not open history ui.
> !not-found.png!
> *From event log directory we can find an event log file size is bigger than 130GB.*
> {code:java}
> hadoop *144149840801* 2017-08-29 14:03 /spark/xxx/log/history/application_1501588539284_1118255_1.lz4.inprogress
> {code}
> *and from jstack and server log we can see replaying task blocked on this event log:*
> *server log:*
> {code:java}
> 2017-10-12,16:00:12,151 INFO org.apache.spark.deploy.history.FsHistoryProvider: Replaying
log path: hdfs://xxx/spark/xxx/log/history/application_1501588539284_1118255_1.lz4.inprogress
> 2017-10-12,16:00:12,167 INFO org.apache.spark.scheduler.ReplayListenerBus: Begin to replay
hdfs://xxx/spark/xxx/log/history/application_1501588539284_1118255_1.lz4.inprogress!
> {code}
> *jstack*
> {code:java}
> "log-replay-executor-0" daemon prio=10 tid=0x00007f0f48014800 nid=0x6160 runnable [0x00007f0f4f6f5000]
>    java.lang.Thread.State: RUNNABLE
>         at net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast(Native Method)
>         at net.jpountz.lz4.LZ4JNIFastDecompressor.decompress(LZ4JNIFastDecompressor.java:37)
>         at org.apache.spark.io.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:205)
>         at org.apache.spark.io.LZ4BlockInputStream.read(LZ4BlockInputStream.java:125)
>         at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
>         at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
>         at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
>         - locked <0x00000005f0096948> (a java.io.InputStreamReader)
>         at java.io.InputStreamReader.read(InputStreamReader.java:184)
>         at java.io.BufferedReader.fill(BufferedReader.java:154)
>         at java.io.BufferedReader.readLine(BufferedReader.java:317)
>         - locked <0x00000005f0096948> (a java.io.InputStreamReader)
>         at java.io.BufferedReader.readLine(BufferedReader.java:382)
>         at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
>         at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:836)
>         at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
>         at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:79)
>         at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
>         at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:776)
>         at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:584)
>         at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$3$$anon$4.run(FsHistoryProvider.scala:464)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message