hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3177) Allow DFSClient to find out and use the CRC type being used for a file.
Date Wed, 22 Aug 2012 13:13:37 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439476#comment-13439476
] 

Tsz Wo (Nicholas), SZE commented on HDFS-3177:
----------------------------------------------

I do not understand the design.  Some questions:

- For append, it makes a lot of sense to keep using the existing checksum type.  What is the
use case for using a different checksum type?

- Suppose the last block is half written with CRC32 in a close file.  Then, the file is re-opened
for append with CRC32C.  Would the block has two checksum types, i.e. first half is CRC32
and the second half is CRC32C?

- Suppose a close file is already using more than one checksum type.  Then, the file is re-opened
for append with dfs.client.append.allow-different-checksum == false.  Which checksum should
it use?  Or should it fail?
                
> Allow DFSClient to find out and use the CRC type being used for a file.
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3177
>                 URL: https://issues.apache.org/jira/browse/HDFS-3177
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node, hdfs client
>    Affects Versions: 0.23.0
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>             Fix For: 2.1.0-alpha, 3.0.0
>
>         Attachments: hdfs-3177-after-hadoop-8239-8240.patch.txt, hdfs-3177-after-hadoop-8239.patch.txt,
hdfs-3177-branch2-trunk.patch.txt, hdfs-3177.patch, hdfs-3177-with-hadoop-8239-8240.patch.txt,
hdfs-3177-with-hadoop-8239-8240.patch.txt, hdfs-3177-with-hadoop-8239-8240.patch.txt, hdfs-3177-with-hadoop-8239.patch.txt
>
>
> To support HADOOP-8060, DFSClient should be able to find out the checksum type being
used for files in hdfs.
> In my prototype, DataTransferProtocol was extended to include the checksum type in the
blockChecksum() response. DFSClient uses it in getFileChecksum() to determin the checksum
type. Also append() can be configured to use the existing checksum type instead of the configured
one.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message