spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-25822) Fix a race condition when releasing a Python worker
Date Wed, 24 Oct 2018 17:26:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-25822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-25822:
------------------------------------

    Assignee: Apache Spark  (was: Shixiong Zhu)

> Fix a race condition when releasing a Python worker
> ---------------------------------------------------
>
>                 Key: SPARK-25822
>                 URL: https://issues.apache.org/jira/browse/SPARK-25822
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.3.2
>            Reporter: Shixiong Zhu
>            Assignee: Apache Spark
>            Priority: Major
>
> There is a race condition when releasing a Python worker. If "ReaderIterator.handleEndOfDataSection"
is not running in the task thread, when a task is early terminated (such as "take(N)"), the
task completion listener may close the worker but "handleEndOfDataSection" can still put the
worker into the worker pool to reuse.
> https://github.com/zsxwing/spark/commit/0e07b483d2e7c68f3b5c3c118d0bf58c501041b7 is a
patch to reproduce this issue.
> I also found a user reported this in the mail list: http://mail-archives.apache.org/mod_mbox/spark-user/201610.mbox/%3CCAAUq=H+YLUEpd23nwvq13Ms5hOStkhX3ao4f4zQV6sgO5zM-xA@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message