spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emilio Dorigatti (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-24034) StopIteration in pyspark mapper results in partial results
Date Fri, 20 Apr 2018 14:37:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Emilio Dorigatti updated SPARK-24034:
-------------------------------------
    Description: 
Consider the following code
{noformat}
def mapper(xx):
     if xx % 2 == 0:
         raise StopIteration()
     else:
         return xx
sc.parallelize(range(100)).map(mapper)collect()
{noformat}

The result I get is `[57, 71, 85]`

I think it happens because {{map }}is implemented in terms of {{mapPartitionsWithIndex}}
using a custom iterator, so the {{StopIteration}} raised by the mapper is handled by that
iterator. I think this should be raised to the user instead

NB: this may be the underlying cause of https://issues.apache.org/jira/browse/SPARK-23754

 

  was:
Consider the following code

    def mapper(xx):
        if xx % 2 == 0:
            raise StopIteration()
        else:
            return xx
    
    sc.parallelize(range(100)).map(mapper)collect()


The result I get is `[57, 71, 85]`

I think it happens because `map` is implemented in terms of  `mapPartitionsWithIndex` using
a custom iterator, so the StopIteration raised by the mapper is handled by that iterator.
I think this should be raised to the user instead

NB: this may be the underlying cause of https://issues.apache.org/jira/browse/SPARK-23754

 


> StopIteration in pyspark mapper results in partial results
> ----------------------------------------------------------
>
>                 Key: SPARK-24034
>                 URL: https://issues.apache.org/jira/browse/SPARK-24034
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.3.0
>            Reporter: Emilio Dorigatti
>            Priority: Major
>
> Consider the following code
> {noformat}
> def mapper(xx):
>      if xx % 2 == 0:
>          raise StopIteration()
>      else:
>          return xx
> sc.parallelize(range(100)).map(mapper)collect()
> {noformat}
> The result I get is `[57, 71, 85]`
> I think it happens because {{map }}is implemented in terms of {{mapPartitionsWithIndex}}
using a custom iterator, so the {{StopIteration}} raised by the mapper is handled by that
iterator. I think this should be raised to the user instead
> NB: this may be the underlying cause of https://issues.apache.org/jira/browse/SPARK-23754
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message