hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8734) Erasure Coding: fix one cell need two packets
Date Wed, 30 Sep 2015 16:02:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937361#comment-14937361
] 

Hudson commented on HDFS-8734:
------------------------------

FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8548/])
HDFS-8734. Erasure Coding: fix one cell need two packets. Contributed by (jing9: rev 0a93712f3b9b36d746577dca5da0f7f09756fcca)
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java


> Erasure Coding: fix one cell need two packets
> ---------------------------------------------
>
>                 Key: HDFS-8734
>                 URL: https://issues.apache.org/jira/browse/HDFS-8734
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Walter Su
>             Fix For: HDFS-7285
>
>         Attachments: HDFS-8734-HDFS-7285.01.patch, HDFS-8734.01.patch
>
>
> The default WritePacketSize is 64k
> Currently default cellSize is 64k
> We hope one cell consumes one packet. In fact it's not.
> By default,
> chunkSize = 516( 512 data + 4 checksum)
> packetSize = 64k
> chunksPerPacket = 126 ( See DFSOutputStream#computePacketChunkSize for details)
> numBytes of data in one packet = 64512
> cellSize = 65536
> When first packet is full ( with 64512 data), there are still 65536 - 64512 = 1024 bytes
left.
> {code}
>         super.writeChunk(bytes, offset, len, checksum, ckoff, cklen);
>         // cell is full and current packet has not been enqueued,
>         if (cellFull && currentPacket != null) {
>           enqueueCurrentPacketFull();
>         }   
> {code}
> When  the last 1024 bytes of the cell was written, we meet {{cellFull}} and create another
packet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message