spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24034) StopIteration in pyspark mapper gives partial results
Date Sat, 21 Apr 2018 17:21:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446896#comment-16446896
] 

Hyukjin Kwon commented on SPARK-24034:
--------------------------------------

Yea, sounds pretty much a duplicate from a quick look. Let's leave this resolved as a duplicate
for now and reopen if we found they are actually not.

> StopIteration in pyspark mapper gives partial results
> -----------------------------------------------------
>
>                 Key: SPARK-24034
>                 URL: https://issues.apache.org/jira/browse/SPARK-24034
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.3.0
>            Reporter: Emilio Dorigatti
>            Priority: Major
>
> Consider the following code
> {noformat}
> def mapper(xx):
>      if xx % 2 == 0:
>          raise StopIteration()
>      else:
>          return xx
> sc.parallelize(range(100)).map(mapper).collect()
> {noformat}
> The result I get is {{[57, 71, 85]}}
> I think it happens because {{map}} is implemented in terms of {{mapPartitionsWithIndex}}
using a custom iterator, so the {{StopIteration}} raised by the mapper is handled by that
iterator. I think this should be raised to the user instead.
> I think I can take care of this, if I am allowed to (first time I contribute, not sure
how it works)
> NB: this may be the underlying cause of https://issues.apache.org/jira/browse/SPARK-23754
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message