hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
Date Tue, 04 Nov 2014 22:54:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14197036#comment-14197036
] 

stack commented on HDFS-7276:
-----------------------------

A few comments on this nice new feature:

I'd suggest you add a release note on how to enable this new feature and list all options
and what they mean. Would help folks get started at least. Also, I'd suggest you might add
a note on how to see if feature is in effect and how to measure how well it is doing (It seems
like enabling DEBUG on ByteArrayManager is way to go.  FYI, my log got 50x bigger in the tests
I'm running enabling DEBUG).

In testing, I seem to have managed to get stuck waiting...

{code}
...
2014-11-04 14:40:57,922 DEBUG [sync.3] util.ByteArrayManager: allocate(65565): count=120668,
aboveThreshold, [131072: 20477/20480, free=1], recycled? true
2014-11-04 14:40:57,922 DEBUG [sync.4] util.ByteArrayManager: allocate(65565): count=120669,
aboveThreshold, [131072: 20478/20480, free=0], recycled? false
2014-11-04 14:40:57,922 DEBUG [sync.2] util.ByteArrayManager: allocate(65565): count=120670,
aboveThreshold, [131072: 20479/20480, free=0], recycled? false
2014-11-04 14:40:57,922 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,922 DEBUG [sync.0] util.ByteArrayManager: allocate(65565): count=120671,
aboveThreshold, [131072: 20479/20480, free=1], recycled? true
2014-11-04 14:40:57,922 DEBUG [sync.1] util.ByteArrayManager: allocate(65565): count=120672,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,922 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,922 DEBUG [sync.1] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,923 DEBUG [sync.3] util.ByteArrayManager: allocate(65565): count=120673,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,923 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,923 DEBUG [sync.3] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,923 DEBUG [sync.4] util.ByteArrayManager: allocate(65565): count=120674,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,923 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,923 DEBUG [sync.4] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,923 DEBUG [sync.0] util.ByteArrayManager: allocate(65565): count=120675,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,924 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,924 DEBUG [sync.0] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,924 DEBUG [sync.2] util.ByteArrayManager: allocate(65565): count=120676,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,924 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,924 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,924 DEBUG [sync.1] util.ByteArrayManager: allocate(65565): count=120677,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,924 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,924 DEBUG [sync.1] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,924 DEBUG [sync.3] util.ByteArrayManager: allocate(65565): count=120678,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,925 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,925 DEBUG [sync.3] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,925 DEBUG [sync.4] util.ByteArrayManager: allocate(65565): count=120679,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
2014-11-04 14:40:57,926 DEBUG [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1075488645_1099513376762]
util.ByteArrayManager: recycle: array.length=131072, [131072: 20480/20480, free=0], notifyAll,
freeQueue.offer, freeQueueSize=1
2014-11-04 14:40:57,926 DEBUG [sync.4] util.ByteArrayManager: wake up: [131072: 20479/20480,
free=1], recycled? true
2014-11-04 14:40:57,926 DEBUG [sync.0] util.ByteArrayManager: allocate(65565): count=120680,
aboveThreshold, [131072: 20480/20480, free=0]: wait ...
{code}

My loading program is stuck here.

Here are my configurations:

{code}
<property>
<name>dfs.client.write.byte-array-manager.enabled</name>
<value>true</value>
</property>

<property>
<name>dfs.client.write.byte-array-manager.count-limit</name>
<value>20480</value>
</property>

<property>
<name>dfs.client.write.byte-array-manager.count-reset-time-period-ms</name>
<value>1000000</value>
</property>
{code}

Have you seen this before?

> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>
>                 Key: HDFS-7276
>                 URL: https://issues.apache.org/jira/browse/HDFS-7276
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.6.0
>
>         Attachments: h7276_20141021.patch, h7276_20141022.patch, h7276_20141023.patch,
h7276_20141024.patch, h7276_20141027.patch, h7276_20141027b.patch, h7276_20141028.patch, h7276_20141029.patch,
h7276_20141029b.patch, h7276_20141030.patch, h7276_20141031.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding
packets could be large.  The byte arrays created by those packets could occupy a lot of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message