hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
Date Mon, 27 Oct 2014 20:00:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185710#comment-14185710

Tsz Wo Nicholas Sze commented on HDFS-7276:

bq. Thinking on it, the patch only limits the outstanding number of Packets if Packets always
allocate same size buffers, right? If the objective is a limit on Packet count, perhaps a
more direct choke on Packet allocations would be more effective?

The objective is to limit the memory usage.  Limiting Packet count may not be desirable since
packets can be small.  They occupy small amount of space even there is a large number of them.

Ideally, we should limit the total size of all arrays.  However, it is unnecessary since non-full
packets are usually small and not frequent.  So, we only limit full packets.

bq. Meantime, I like the buffer reuse that this patch brings.


> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>                 Key: HDFS-7276
>                 URL: https://issues.apache.org/jira/browse/HDFS-7276
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h7276_20141021.patch, h7276_20141022.patch, h7276_20141023.patch,
> When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding
packets could be large.  The byte arrays created by those packets could occupy a lot of memory.

This message was sent by Atlassian JIRA

View raw message