spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Davidson (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-1866) Closure cleaner does not null shadowed fields when outer scope is referenced
Date Sat, 17 May 2014 03:14:14 GMT
Aaron Davidson created SPARK-1866:
-------------------------------------

             Summary: Closure cleaner does not null shadowed fields when outer scope is referenced
                 Key: SPARK-1866
                 URL: https://issues.apache.org/jira/browse/SPARK-1866
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.0.0
            Reporter: Aaron Davidson
            Priority: Critical
             Fix For: 1.1.0, 1.0.1


Take the following example:
{code}
val x = 5
val instances = new org.apache.hadoop.fs.Path("/") /* non-serializable */
sc.parallelize(0 until 10).map { _ =>
  val instances = 3
  (instances, x)
}.collect
{code}

This produces a "java.io.NotSerializableException: org.apache.hadoop.fs.Path", despite the
fact that the outer instances is not actually used within the closure. If you change the name
of the outer variable instances to something else, the code executes correctly, indicating
that it is the fact that the two variables share a name that causes the issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message