hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
Date Tue, 04 Nov 2014 23:27:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14197115#comment-14197115
] 

stack commented on HDFS-7276:
-----------------------------

This seems to be holding up progress:
{code}
"sync.0" #35 prio=5 os_prio=0 tid=0x00007f5c281d2000 nid=0x6f7e in Object.wait() [0x00007f5bf71e5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.hadoop.hdfs.util.ByteArrayManager$FixedLengthManager.allocate(ByteArrayManager.java:174)
        - locked <0x00000002280915c0> (a org.apache.hadoop.hdfs.util.ByteArrayManager$FixedLengthManager)
        at org.apache.hadoop.hdfs.util.ByteArrayManager$Impl.newByteArray(ByteArrayManager.java:374)
        at org.apache.hadoop.hdfs.DFSOutputStream.createPacket(DFSOutputStream.java:192)
        at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1868)
        - locked <0x00000002279bbc40> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:206)
        at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163)
        - locked <0x00000002279bbc40> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSOutputSummer.flush(FSOutputSummer.java:182)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.DataOutputStream.flush(DataOutputStream.java:123)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:167)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1302)
        at java.lang.Thread.run(Thread.java:745)
{code}

> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>
>                 Key: HDFS-7276
>                 URL: https://issues.apache.org/jira/browse/HDFS-7276
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.6.0
>
>         Attachments: h7276_20141021.patch, h7276_20141022.patch, h7276_20141023.patch,
h7276_20141024.patch, h7276_20141027.patch, h7276_20141027b.patch, h7276_20141028.patch, h7276_20141029.patch,
h7276_20141029b.patch, h7276_20141030.patch, h7276_20141031.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding
packets could be large.  The byte arrays created by those packets could occupy a lot of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message