spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-16930) ApplicationMaster's code that waits for SparkContext is race-prone
Date Wed, 17 Aug 2016 18:13:22 GMT

     [ https://issues.apache.org/jira/browse/SPARK-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin resolved SPARK-16930.
------------------------------------
       Resolution: Fixed
         Assignee: Marcelo Vanzin
    Fix Version/s: 2.1.0

> ApplicationMaster's code that waits for SparkContext is race-prone
> ------------------------------------------------------------------
>
>                 Key: SPARK-16930
>                 URL: https://issues.apache.org/jira/browse/SPARK-16930
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>             Fix For: 2.1.0
>
>
> While taking a look at SPARK-15937 and checking if there's something wrong with the code,
I noticed two races that explain the behavior.
> Because they're really narrow races, I'm a little wary of declaring them the cause of
that bug. Also because the logs posted there don't really explain what went wrong (and don't
really look like a SparkContext was run at all).
> The races I found are:
> - it's possible, but very unlikely, for an application to instantiate a SparkContext
and stop it before the AM enters the loop where it checks for the instance.
> - it's possible, but very unlikely, for an application to stop the SparkContext after
the AM is already waiting for one, has been notified of its creation, but hasn't yet stored
the SparkContext reference in a local variable.
> I'll fix those and clean up the code a bit in the process.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message