hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
Date Fri, 31 Oct 2014 19:39:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192322#comment-14192322

Tsz Wo Nicholas Sze commented on HDFS-7276:

bq. Can't we simply create those 12 or so managers immediately, and avoid all the "create
a new manager if statistics say so" logic?

ByteArrayManager is a generic class.  So it may not necessarily be 12.

174	        wait(1000);

Why does this wait have a timeout? There isn't anything that we're polling here.

...    synchronized int recycle(byte[] array) {

Shouldn't this simply be "notify"? There is no point in waking up all waiters, because only
one of them is going to get the buffer we're recycling.
Both the wait time out and the notifyAll are for preventing deadlock.  After some thought,
only one of them is needed.  I choose to keep notifyAll since the timeout value is hard to

> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>                 Key: HDFS-7276
>                 URL: https://issues.apache.org/jira/browse/HDFS-7276
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h7276_20141021.patch, h7276_20141022.patch, h7276_20141023.patch,
h7276_20141024.patch, h7276_20141027.patch, h7276_20141027b.patch, h7276_20141028.patch, h7276_20141029.patch,
h7276_20141029b.patch, h7276_20141030.patch
> When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding
packets could be large.  The byte arrays created by those packets could occupy a lot of memory.

This message was sent by Atlassian JIRA

View raw message