hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Thomas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)
Date Wed, 27 Aug 2014 06:39:58 GMT

     [ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

James Thomas updated HDFS-6865:

    Attachment: HDFS-6865.6.patch

[~tlipcon], thanks for the review. Made all of your suggested changes except the one about
calling {{writeChunk}} with larger buffers -- if you look at {{DFSOutputStream#writeChunk}},
you'll see that it has some assertions that check that only a single chunk has been passed
in. It also calls {{Packet#writeData}} and {{Packet#writeChecksum}}, which expect that the
written data will not overflow the packet. So we would need some logic that checks whether
the write will overflow the packet and, if so, writes only enough to fill the packet, queues
it, and then allocates a new packet and writes the remainder to that. Sounds like this is
best left for later.

> Byte array native checksumming on client side (HDFS changes)
> ------------------------------------------------------------
>                 Key: HDFS-6865
>                 URL: https://issues.apache.org/jira/browse/HDFS-6865
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client, performance
>            Reporter: James Thomas
>            Assignee: James Thomas
>         Attachments: HDFS-6865.2.patch, HDFS-6865.3.patch, HDFS-6865.4.patch, HDFS-6865.5.patch,
HDFS-6865.6.patch, HDFS-6865.patch
> Refactor FSOutputSummer to buffer data and use the native checksum calculation functionality
introduced in HADOOP-10975.

This message was sent by Atlassian JIRA

View raw message