spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Resolved] (SPARK-11058) failed spark job reports on YARN as successful
Date Fri, 16 Jun 2017 23:35:00 GMT


Marcelo Vanzin resolved SPARK-11058.
    Resolution: Not A Problem

This is a limitation of client mode in Spark. The YARN application always shows up as successful.
The {{SparkContext}} API is not rich enough for the driver to tell the AM whether to succeed
or fail the YARN application.

> failed spark job reports on YARN as successful
> ----------------------------------------------
>                 Key: SPARK-11058
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.3.0
>         Environment: CDH 5.4
>            Reporter: Lan Jiang
>            Priority: Minor
> I have a spark batch job running on CDH5.4 + Spark 1.3.0. Job is submitted in “yarn-client”
mode. The job itself failed due to YARN kills several executor containers because the containers
exceeded the memory limit posed by YARN. However, when I went to the YARN resource manager
site, it displayed the job as successful. I found there was an issue reported in JIRA,
but it says it was fixed in Spark 1.2. On Spark history server, it shows the job as “Incomplete”.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message