spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zenglinxi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21223) Thread-safety issue in FsHistoryProvider
Date Thu, 29 Jun 2017 03:38:02 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

zenglinxi updated SPARK-21223:
------------------------------
    Attachment: historyserver_jstack.txt

BTW, this cause an infinite loop problem when we restart historyserver and replaying event
logs of spark apps.

> Thread-safety issue in FsHistoryProvider 
> -----------------------------------------
>
>                 Key: SPARK-21223
>                 URL: https://issues.apache.org/jira/browse/SPARK-21223
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.1
>            Reporter: zenglinxi
>         Attachments: historyserver_jstack.txt
>
>
> Currently, Spark HistoryServer use a HashMap named fileToAppInfo in class FsHistoryProvider
to store the map of eventlog path and attemptInfo. 
> When use ThreadPool to Replay the log files in the list and merge the list of old applications
with new ones, multi thread may update fileToAppInfo at the same time, which may cause Thread-safety
issues.
> {code:java}
> for (file <- logInfos) {
>        tasks += replayExecutor.submit(new Runnable {
>         override def run(): Unit = mergeApplicationListing(file)
>      })
>  }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message