Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm
Precedence: bulk
From: HyukjinKwon <git@git.apache.org>
To: reviews@spark.apache.org
Reply-To: reviews@spark.apache.org
References: <git-pr-18537-spark@git.apache.org>
In-Reply-To: <git-pr-18537-spark@git.apache.org>
Subject: [GitHub] spark pull request #18537: [SPARK-20609][MLLIB][TEST]manually cleared 'spark...
Content-Type: text/plain
Message-Id: <20170705114254.A2C95DFC25@git1-us-west.apache.org>
Date: Wed,  5 Jul 2017 11:42:54 +0000 (UTC)
archived-at: Wed, 05 Jul 2017 11:42:57 -0000

Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18537#discussion_r125620580
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala ---
    @@ -777,15 +778,34 @@ class ALSSuite
       }
     }
     
    -class ALSCleanerSuite extends SparkFunSuite {
    +class ALSCleanerSuite extends SparkFunSuite with BeforeAndAfterEach {
    +
    +  private var localDir: File = _
    --- End diff --
    
    Just to make sure the issue, I debugged as below:
    
    **Before**
    
    ```diff
    @@ -785,7 +785,10 @@ class ALSCleanerSuite extends SparkFunSuite {
         def getAllFiles: Set[File] =
           FileUtils.listFiles(localDir, TrueFileFilter.INSTANCE, TrueFileFilter.INSTANCE).asScala.toSet
         try {
    +      println(s"Before local dir : ${Utils.getLocalDir(conf)}")
           conf.set("spark.local.dir", localDir.getAbsolutePath)
    +      println(s"Actual local dir : ${Utils.getLocalDir(conf)}")
    +      println(s"Expected local dir root : ${localDir.getAbsolutePath}")
           val sc = new SparkContext("local[2]", "test", conf)
           try {
             sc.setCheckpointDir(checkpointDir.getAbsolutePath)
    @@ -854,77 +857,77 @@ class ALSCleanerSuite extends SparkFunSuite {
    ```
    
    
    ```
    Before local dir : /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-bd7219ae-e205-49d4-bebf-2970ae4bab4d
    Actual local dir : /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-bd7219ae-e205-49d4-bebf-2970ae4bab4d
    Expected local dir root : /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-167efe6b-ec4a-4fda-80f3-3dc970439d51
    ```
    
    
    **After**
    
    ```diff
    @@ -785,7 +785,11 @@ class ALSCleanerSuite extends SparkFunSuite {
         def getAllFiles: Set[File] =
           FileUtils.listFiles(localDir, TrueFileFilter.INSTANCE, TrueFileFilter.INSTANCE).asScala.toSet
         try {
    +      println(s"Before local dir : ${Utils.getLocalDir(conf)}")
           conf.set("spark.local.dir", localDir.getAbsolutePath)
    +      Utils.clearLocalRootDirs()
    +      println(s"Actual local dir : ${Utils.getLocalDir(conf)}")
    +      println(s"Expected local dir root : ${localDir.getAbsolutePath}")
           val sc = new SparkContext("local[2]", "test", conf)
           try {
             sc.setCheckpointDir(checkpointDir.getAbsolutePath)
    @@ -854,77 +858,77 @@ class ALSCleanerSuite extends SparkFunSuite {
    ```
    
    
    ```
    Before local dir : /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-f5f2e39f-b4da-48aa-b6e9-f26b5b80c53e
    Actual local dir : /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-6e4ead48-2a0f-49b4-8cd7-4a81803aafee/spark-22dff601-c972-4a31-8357-cb595346487b
    Expected local dir root : /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-6e4ead48-2a0f-49b4-8cd7-4a81803aafee
    ```
    
    I just double checked `getLocalDir` is called with initiation of `SparkContext` as below:
    
    <img width="273" alt="2017-07-05 8 30 28" src="https://user-images.githubusercontent.com/6477701/27862873-6d434778-61c2-11e7-89b8-5d4c6a8506e2.png">
    
    <img width="478" alt="2017-07-05 8 30 21" src="https://user-images.githubusercontent.com/6477701/27862876-6e221fe8-61c2-11e7-89d1-f566ccf9f7f3.png">


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org