hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Verdi March <verdi.ma...@gmail.com>
Subject Calculation of packet size in DFSOutputStream
Date Wed, 26 Sep 2012 00:31:43 GMT
Hi,

I'd like to ask about the calculation of packet size, as implemented in
DFSClient.DFSOutputStream (Hadoop v1.0.3). My apology if this is not
the appropriate list.

1.  private void computePacketChunkSize(int psize, int csize) {
2.    int chunkSize = csize + checksum.getChecksumSize();
3.    int n = DataNode.PKT_HEADER_LEN + SIZE_OF_INTEGER;
4.    chunksPerPacket = Math.max((psize - n + chunkSize-1)/chunkSize, 1);
5.    packetSize = n + chunkSize*chunksPerPacket;
6.    if (LOG.isDebugEnabled()) {
7.      LOG.debug("computePacketChunkSize: src=" + src +
8.                ", chunkSize=" + chunkSize +
9.                ", chunksPerPacket=" + chunksPerPacket +
10.               ", packetSize=" + packetSize);
11.   }
12. }

My question is re. line-4. The debug log shows that the packet header
(PKT_HEADER_LEN) is not part of the payload (64KB by default), so why
still substract psize with n instead of with SIZE_OF_INTEGER?

I'm not entirely familiar with the intricacies of HDFS, so am hoping
for some pointers or directions.

Regards,
Verdi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message