spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Erlandson (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-4514) ComplexFutureAction does not preserve job group IDs
Date Thu, 20 Nov 2014 19:27:34 GMT
Erik Erlandson created SPARK-4514:
-------------------------------------

             Summary: ComplexFutureAction does not preserve job group IDs
                 Key: SPARK-4514
                 URL: https://issues.apache.org/jira/browse/SPARK-4514
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
            Reporter: Erik Erlandson


The current job group id of a Spark context is stored in the {{localProperties}} member value.
  This data structure is designed to be thread local, and its settings are not preserved when
{{ComplexFutureAction}} instantiates a new {{Future}}.  

One consequence of this is that {{takeAsync()}} does not behave in the same way as other async
actions, e.g. {{countAsync()}}.  For example, this test (if copied into StatusTrackerSuite.scala),
will fail, because {{"my-job-group2"}} is not propagated to the Future which actually instantiates
the job:

{code:java}
  test("getJobIdsForGroup() with takeAsync()") {
    sc = new SparkContext("local", "test", new SparkConf(false))
    sc.setJobGroup("my-job-group2", "description")
    sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq.empty)
    val firstJobFuture = sc.parallelize(1 to 1000, 1).takeAsync(1)
    val firstJobId = eventually(timeout(10 seconds)) {
      firstJobFuture.jobIds.head
    }
    eventually(timeout(10 seconds)) {
      sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq(firstJobId))
    }
  }
{code}

It also impacts current PR for SPARK-1021, which involves additional uses of {{ComplexFutureAction}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message