Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 20 Nov 2014 19:27:34 +0000 (UTC)
From: "Erik Erlandson (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12756776.1416511629000.546861.1416511654447@Atlassian.JIRA>
In-Reply-To: <JIRA.12756776.1416511629000@Atlassian.JIRA>
References: <JIRA.12756776.1416511629000@Atlassian.JIRA>
 <JIRA.12756776.1416511629385@arcas>
Subject: [jira] [Created] (SPARK-4514) ComplexFutureAction does not preserve
 job group IDs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Erik Erlandson created SPARK-4514:
-------------------------------------

             Summary: ComplexFutureAction does not preserve job group IDs
                 Key: SPARK-4514
                 URL: https://issues.apache.org/jira/browse/SPARK-4514
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
            Reporter: Erik Erlandson


The current job group id of a Spark context is stored in the {{localProperties}} member value.   This data structure is designed to be thread local, and its settings are not preserved when {{ComplexFutureAction}} instantiates a new {{Future}}.  

One consequence of this is that {{takeAsync()}} does not behave in the same way as other async actions, e.g. {{countAsync()}}.  For example, this test (if copied into StatusTrackerSuite.scala), will fail, because {{"my-job-group2"}} is not propagated to the Future which actually instantiates the job:

{code:java}
  test("getJobIdsForGroup() with takeAsync()") {
    sc = new SparkContext("local", "test", new SparkConf(false))
    sc.setJobGroup("my-job-group2", "description")
    sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq.empty)
    val firstJobFuture = sc.parallelize(1 to 1000, 1).takeAsync(1)
    val firstJobId = eventually(timeout(10 seconds)) {
      firstJobFuture.jobIds.head
    }
    eventually(timeout(10 seconds)) {
      sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq(firstJobId))
    }
  }
{code}

It also impacts current PR for SPARK-1021, which involves additional uses of {{ComplexFutureAction}}.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org