spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From viirya <...@git.apache.org>
Subject [GitHub] spark pull request #15178: [SPARK-17556][SQL] Executor side broadcast for br...
Date Thu, 29 Sep 2016 01:49:36 GMT
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15178#discussion_r81052440
  
    --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentExecutorBroadcast.scala
---
    @@ -0,0 +1,174 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.broadcast
    +
    +import java.io.ObjectOutputStream
    +
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +import scala.util.Random
    +
    +import org.apache.spark._
    +import org.apache.spark.internal.Logging
    +import org.apache.spark.storage.{BlockId, BlockResult, BroadcastBlockId, RDDBlockId,
StorageLevel}
    +import org.apache.spark.util.Utils
    +
    +/**
    + * A BitTorrent-like implementation of [[org.apache.spark.broadcast.Broadcast]].
    + *
    + * Different to [[TorrentBroadcast]], this implementation doesn't divide the object to
broadcast.
    + * In contrast, this implementation performs broadcast on executor side for a RDD. So
the results
    + * of the RDD does not need to collect first back to the driver before broadcasting.
    + *
    + * The mechanism is as follows:
    + *
    + * On each executor, the executor first attempts to fetch the object from its BlockManager.
If
    + * it doesn not exist, it then uses remote fetches to fetch the blocks of the RDD from
other
    + * executors if available. Once it gets the blocks, it puts the blocks in its own BlockManager,
    + * ready for other executors to fetch from.
    + *
    + * @tparam T The type of the element of RDD to be broadcasted.
    + * @tparam U The type of object transformed from the collection of elements of the RDD.
    + *
    + * @param numBlocks Total number of blocks this broadcast variable contains.
    + * @param rddId The id of the RDD to be broadcasted on executors.
    + * @param mode The [[BroadcastMode]] object used to transform the result of RDD to the
object which
    + *             will be stored in the [[BlockManager]].
    + * @param id A unique identifier for the broadcast variable.
    + */
    +private[spark] class TorrentExecutorBroadcast[T: ClassTag, U: ClassTag](
    +    numBlocks: Int,
    +    rddId: Int,
    +    mode: BroadcastMode[T],
    +    id: Long) extends Broadcast[U](id) with Logging with Serializable {
    +
    +  /**
    +   * Value of the broadcast object on executors. This is reconstructed by [[readBroadcastBlock]],
    +   * which builds this value by reading blocks from other executors.
    +   */
    +  @transient private lazy val _value: U = readBroadcastBlock()
    +
    +  private val broadcastId = BroadcastBlockId(id)
    +
    +  override protected def getValue() = {
    +    _value
    +  }
    +
    +  /** Fetch torrent blocks from other executors. */
    +  private def readBlocks(): Array[T] = {
    +    // Fetch chunks of data. Note that all these chunks are stored in the BlockManager
and reported
    +    // to the driver, so other executors can pull these chunks from this executor as
well.
    +    val blocks = new Array[Array[T]](numBlocks)
    +    val bm = SparkEnv.get.blockManager
    +
    +    for (pid <- Random.shuffle(Seq.range(0, numBlocks))) {
    +      val pieceId = RDDBlockId(rddId, pid)
    +      // First try getLocalValues because there is a chance that previous attempts to
fetch the
    +      // broadcast blocks have already fetched some of the blocks. In that case, some
blocks
    +      // would be available locally (on this executor).
    +      bm.getLocalValues(pieceId) match {
    +        case Some(block: BlockResult) =>
    +          blocks(pid) = block.data.asInstanceOf[Iterator[T]].toArray
    +        case None =>
    +          bm.get[T](pieceId) match {
    +            case Some(b) =>
    +              val data = b.data.asInstanceOf[Iterator[T]].toArray
    +              // We found the block from remote executors' BlockManager, so put the block
    +              // in this executor's BlockManager.
    +              if (!bm.putIterator(pieceId, data.toIterator,
    --- End diff --
    
    One solution might be to store the fetched RDD pieces with broadcast piece ID, so in unpersist
we can remove all the fetched pieces. However, then we must consider fetch both RDD piece
IDs broadcast IDs from other executors under the BitTorrent-like approach. Thus I would prefer
the above way and let current cleaning mechanism do its work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message