spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brennon York (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-4170) Closure problems when running Scala app that "extends App"
Date Thu, 27 Nov 2014 06:49:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14227319#comment-14227319
] 

Brennon York edited comment on SPARK-4170 at 11/27/14 6:48 AM:
---------------------------------------------------------------

[~srowen] this bug seems to be an issue with the way Scala has been defined and the differences
between what happens at runtime versus compile time with respect to the way App leverages
the [delayedInit function|http://www.scala-lang.org/api/2.11.1/index.html#scala.App].

I tried to replicate the issue on my local machine under both compile time and runtime with
only the latter producing the issue (as expected through the Scala documentation). The former
was tested by creating a simple application, compiled with sbt, and executed while the latter
was setup within the spark-shell REPL. I'm wondering if we can't close this issue and just
provide a bit of documentation somewhere to reference that, when building even simple Spark
apps, extending the App interface will result in delayed initialization and, likely, set null
values within that closure. Thoughts?


was (Author: boyork):
[~srowen] this bug seems to be an issue with the way Scala has been defined and the differences
between what happens at runtime versus compile time with respect to the way {code}App{code}
leverages the [{code}delayedInit{code} function|http://www.scala-lang.org/api/2.11.1/index.html#scala.App].

I tried to replicate the issue on my local machine under both compile time and runtime with
only the latter producing the issue (as expected through the Scala documentation). The former
was tested by creating a simple application, compiled with sbt, and executed while the latter
was setup within the {code}spark-shell{code} REPL. I'm wondering if we can't close this issue
and just provide a bit of documentation somewhere to reference that, when building even simple
Spark apps, extending the {code}App{code} interface will result in delayed initialization
and, likely, set null values within that closure. Thoughts?

> Closure problems when running Scala app that "extends App"
> ----------------------------------------------------------
>
>                 Key: SPARK-4170
>                 URL: https://issues.apache.org/jira/browse/SPARK-4170
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0
>            Reporter: Sean Owen
>            Priority: Minor
>
> Michael Albert noted this problem on the mailing list (http://apache-spark-user-list.1001560.n3.nabble.com/BUG-when-running-as-quot-extends-App-quot-closures-don-t-capture-variables-td17675.html):
> {code}
> object DemoBug extends App {
>     val conf = new SparkConf()
>     val sc = new SparkContext(conf)
>     val rdd = sc.parallelize(List("A","B","C","D"))
>     val str1 = "A"
>     val rslt1 = rdd.filter(x => { x != "A" }).count
>     val rslt2 = rdd.filter(x => { str1 != null && x != "A" }).count
>     
>     println("DemoBug: rslt1 = " + rslt1 + " rslt2 = " + rslt2)
> }
> {code}
> This produces the output:
> {code}
> DemoBug: rslt1 = 3 rslt2 = 0
> {code}
> If instead there is a proper "main()", it works as expected.
> I also this week noticed that in a program which "extends App", some values were inexplicably
null in a closure. When changing to use main(), it was fine.
> I assume there is a problem with variables not being added to the closure when main()
doesn't appear in the standard way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message