hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8734) Erasure Coding: one cell need two packets
Date Wed, 08 Jul 2015 10:04:04 GMT
Walter Su created HDFS-8734:
-------------------------------

             Summary: Erasure Coding: one cell need two packets
                 Key: HDFS-8734
                 URL: https://issues.apache.org/jira/browse/HDFS-8734
             Project: Hadoop HDFS
          Issue Type: Sub-task
            Reporter: Walter Su
            Assignee: Walter Su


The default WritePacketSize is 64k
Currently default cellSize is 64k

We hope one cell consumes one packet. In fact it's not.

By default,
chunkSize = 516( 512 data + 4 checksum)
packetSize = 64k
chunksPerPacket = 126 ( See DFSOutputStream#computePacketChunkSize for details)
numBytes of data in one packet = 64512
cellSize = 65536

When first packet is full ( with 64512 data), there are still 65536 - 64512 = 1024 bytes left.
{code}
        super.writeChunk(bytes, offset, len, checksum, ckoff, cklen);

        // cell is full and current packet has not been enqueued,
        if (cellFull && currentPacket != null) {
          enqueueCurrentPacketFull();
        }   
{code}
When  the last 1024 bytes of the cell was written, we meet {{cellFull}} and create another
packet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message