spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Erlandson (JIRA)" <>
Subject [jira] [Created] (SPARK-4514) ComplexFutureAction does not preserve job group IDs
Date Thu, 20 Nov 2014 19:27:34 GMT
Erik Erlandson created SPARK-4514:

             Summary: ComplexFutureAction does not preserve job group IDs
                 Key: SPARK-4514
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
            Reporter: Erik Erlandson

The current job group id of a Spark context is stored in the {{localProperties}} member value.
  This data structure is designed to be thread local, and its settings are not preserved when
{{ComplexFutureAction}} instantiates a new {{Future}}.  

One consequence of this is that {{takeAsync()}} does not behave in the same way as other async
actions, e.g. {{countAsync()}}.  For example, this test (if copied into StatusTrackerSuite.scala),
will fail, because {{"my-job-group2"}} is not propagated to the Future which actually instantiates
the job:

  test("getJobIdsForGroup() with takeAsync()") {
    sc = new SparkContext("local", "test", new SparkConf(false))
    sc.setJobGroup("my-job-group2", "description")
    sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq.empty)
    val firstJobFuture = sc.parallelize(1 to 1000, 1).takeAsync(1)
    val firstJobId = eventually(timeout(10 seconds)) {
    eventually(timeout(10 seconds)) {
      sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq(firstJobId))

It also impacts current PR for SPARK-1021, which involves additional uses of {{ComplexFutureAction}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message