hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
Date Thu, 13 Nov 2014 21:03:34 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tsz Wo Nicholas Sze updated HDFS-7276:
    Release Note: 
The following configuration properties are added.

- dfs.client.write.byte-array-manager.enabled:
for enabling/disabling byte array manger.  Default is false.

- dfs.client.write.byte-array-manager.count-threshold:
The count threshold for each array length so that a manager is created only after the allocation
count exceeds the threshold.  In other words, the particular array length is not managed until
the allocation count exceeds the threshold.  Default is 128.

- dfs.client.write.byte-array-manager.count-limit:
The maximum number of arrays allowed for each array length.  Default is 2048.

- dfs.client.write.byte-array-manager.count-reset-time-period-ms:
The time period in milliseconds that the allocation count for each array length is reset to
zero if there is no increment.  Default is 10,000ms, i.e. 10 seconds.

Added release note.

> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>                 Key: HDFS-7276
>                 URL: https://issues.apache.org/jira/browse/HDFS-7276
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.6.0
>         Attachments: h7276_20141021.patch, h7276_20141022.patch, h7276_20141023.patch,
h7276_20141024.patch, h7276_20141027.patch, h7276_20141027b.patch, h7276_20141028.patch, h7276_20141029.patch,
h7276_20141029b.patch, h7276_20141030.patch, h7276_20141031.patch
> When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding
packets could be large.  The byte arrays created by those packets could occupy a lot of memory.

This message was sent by Atlassian JIRA

View raw message