spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jerryshao <...@git.apache.org>
Subject [GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...
Date Wed, 18 Jul 2018 04:51:38 GMT
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21440#discussion_r203250619
  
    --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala ---
    @@ -166,6 +170,34 @@ private[spark] class ChunkedByteBuffer(var chunks: Array[ByteBuffer])
{
     
     }
     
    +object ChunkedByteBuffer {
    +  // TODO eliminate this method if we switch BlockManager to getting InputStreams
    +  def fromManagedBuffer(data: ManagedBuffer, maxChunkSize: Int): ChunkedByteBuffer =
{
    +    data match {
    +      case f: FileSegmentManagedBuffer =>
    +        map(f.getFile, maxChunkSize, f.getOffset, f.getLength)
    +      case other =>
    +        new ChunkedByteBuffer(other.nioByteBuffer())
    +    }
    +  }
    +
    +  def map(file: File, maxChunkSize: Int, offset: Long, length: Long): ChunkedByteBuffer
= {
    +    Utils.tryWithResource(new FileInputStream(file).getChannel()) { channel =>
    --- End diff --
    
    I've already updated some of them in SPARK-21475 in shuffle related code path, but not
all of them which are not so critical.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message