spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From brad-kaiser <...@git.apache.org>
Subject [GitHub] spark pull request #19836: [SPARK-22618][CORE] Catch exception in removeRDD ...
Date Wed, 06 Dec 2017 14:48:06 GMT
Github user brad-kaiser commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19836#discussion_r155255972
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
---
    @@ -159,11 +160,18 @@ class BlockManagerMasterEndpoint(
         // Ask the slaves to remove the RDD, and put the result in a sequence of Futures.
         // The dispatcher is used as an implicit argument into the Future sequence construction.
         val removeMsg = RemoveRdd(rddId)
    -    Future.sequence(
    -      blockManagerInfo.values.map { bm =>
    -        bm.slaveEndpoint.ask[Int](removeMsg)
    -      }.toSeq
    -    )
    +
    +    val handleRemoveRddException: PartialFunction[Throwable, Int] = {
    +      case e: IOException =>
    +        logWarning(s"Error trying to remove RDD $rddId", e)
    +        0 // zero blocks were removed
    +    }
    +
    +    val futures = blockManagerInfo.values.map { bm =>
    +      bm.slaveEndpoint.ask[Int](removeMsg).recover(handleRemoveRddException)
    --- End diff --
    
    Ok, I updated this. Thanks.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message