spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21527) Use buffer limit in order to take advantage of JAVA NIO Util's buffercache
Date Mon, 25 Sep 2017 12:09:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178919#comment-16178919
] 

Apache Spark commented on SPARK-21527:
--------------------------------------

User 'caneGuy' has created a pull request for this issue:
https://github.com/apache/spark/pull/18730

> Use buffer limit in order to take advantage of  JAVA NIO Util's buffercache
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-21527
>                 URL: https://issues.apache.org/jira/browse/SPARK-21527
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: zhoukang
>            Assignee: zhoukang
>             Fix For: 2.3.0
>
>
> Right now, ChunkedByteBuffer#writeFully do not slice bytes first.We observe code in java
nio Util below:
> {code:java}
> public static ByteBuffer More ...getTemporaryDirectBuffer(int size) {
>         BufferCache cache = bufferCache.get();
>         ByteBuffer buf = cache.get(size);
>         if (buf != null) {
>             return buf;
>         } else {
>             // No suitable buffer in the cache so we need to allocate a new
>             // one. To avoid the cache growing then we remove the first
>             // buffer from the cache and free it.
>             if (!cache.isEmpty()) {
>                 buf = cache.removeFirst();
>                 free(buf);
>             }
>             return ByteBuffer.allocateDirect(size);
>         }
>     }
> {code}
> If we slice first with a fixed size, we can use buffer cache and only need to allocate
at the first write call.
> Since we allocate new buffer, we can not control the free time of this buffer.This once
cause memory issue in our production cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message