spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joshro...@apache.org
Subject spark git commit: [SPARK-4714] BlockManager.dropFromMemory() should check whether block has been removed after synchronizing on BlockInfo instance.
Date Wed, 17 Dec 2014 20:17:06 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.2 51081e42b -> 0ebbccb87


[SPARK-4714] BlockManager.dropFromMemory() should check whether block has been removed after
synchronizing on BlockInfo instance.

After synchronizing on the `info` lock in the `removeBlock`/`dropOldBlocks`/`dropFromMemory`
methods in BlockManager, the block that `info` represented may have already removed.

The three methods have the same logic to get the `info` lock:
```
   info = blockInfo.get(id)
   if (info != null) {
     info.synchronized {
       // do something
     }
   }
```

So, there is chance that when a thread enters the `info.synchronized` block, `info` has already
been removed from the `blockInfo` map by some other thread who entered `info.synchronized`
first.

The `removeBlock` and `dropOldBlocks` methods are idempotent, so it's safe for them to run
on blocks that have already been removed.
But in `dropFromMemory` it may be problematic since it may drop block data which already removed
into the diskstore, and this calls data store operations that are not designed to handle missing
blocks.

This patch fixes this issue by adding a check to `dropFromMemory` to test whether blocks have
been removed by a racing thread.

Author: hushan[胡珊] <hushan@xiaomi.com>

Closes #3574 from suyanNone/refine-block-concurrency and squashes the following commits:

edb989d [hushan[胡珊]] Refine code style and comments position
55fa4ba [hushan[胡珊]] refine code
e57e270 [hushan[胡珊]] add check info is already remove or not while having gotten info.syn

(cherry picked from commit 30dca924df0efbdc1b638fa7c705fe8743570783)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0ebbccb8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0ebbccb8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0ebbccb8

Branch: refs/heads/branch-1.2
Commit: 0ebbccb872a7379a63f798c67f5da692325ea4d0
Parents: 51081e4
Author: hushan[胡珊] <hushan@xiaomi.com>
Authored: Tue Dec 9 15:11:20 2014 -0800
Committer: Josh Rosen <joshrosen@databricks.com>
Committed: Wed Dec 17 12:17:01 2014 -0800

----------------------------------------------------------------------
 core/src/main/scala/org/apache/spark/storage/BlockManager.scala | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/0ebbccb8/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
index 308c59e..d7b184f 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
@@ -1014,8 +1014,10 @@ private[spark] class BlockManager(
           // If we get here, the block write failed.
           logWarning(s"Block $blockId was marked as failure. Nothing to drop")
           return None
+        } else if (blockInfo.get(blockId).isEmpty) {
+          logWarning(s"Block $blockId was already dropped.")
+          return None
         }
-
         var blockIsUpdated = false
         val level = info.level
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message