spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Jin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-22655) Fail task instead of complete task silently in PythonRunner during shutdown
Date Wed, 29 Nov 2017 22:05:00 GMT
Li Jin created SPARK-22655:
------------------------------

             Summary: Fail task instead of complete task silently in PythonRunner during shutdown
                 Key: SPARK-22655
                 URL: https://issues.apache.org/jira/browse/SPARK-22655
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.2.0, 2.1.0, 2.0.2
            Reporter: Li Jin


We have observed in our production environment that during Spark shutdown, if there are some
active tasks, sometimes they will complete with incorrect results. We've tracked down the
issue to a PythonRunner where it is returning partial result instead of throwing exception
during Spark shutdown. 

I think the better way to handle this is to have these tasks fail instead of complete with
partial results (complete with partial is always bad IMHO)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message