spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-22655) Fail task instead of complete task silently in PythonRunner during shutdown
Date Fri, 08 Dec 2017 11:46:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyukjin Kwon reassigned SPARK-22655:
------------------------------------

    Assignee: Li Jin

> Fail task instead of complete task silently in PythonRunner during shutdown
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-22655
>                 URL: https://issues.apache.org/jira/browse/SPARK-22655
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.0.2, 2.1.0, 2.2.0
>            Reporter: Li Jin
>            Assignee: Li Jin
>             Fix For: 2.3.0
>
>
> We have observed in our production environment that during Spark shutdown, if there are
some active tasks, sometimes they will complete with incorrect results. We've tracked down
the issue to a PythonRunner where it is returning partial result instead of throwing exception
during Spark shutdown. 
> I think the better way to handle this is to have these tasks fail instead of complete
with partial results (complete with partial is always bad IMHO)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message