hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8240) Allow users to specify a checksum type on create()
Date Wed, 14 Nov 2012 20:26:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497410#comment-13497410
] 

Kihwal Lee commented on HADOOP-8240:
------------------------------------

bq. Since there is no meta file content updated with CRC, it can not verify CRCs right?

Checksum verification won't be done, but when it tries to read the data it may detect latent
sector errors, which are becoming more of an issue as disk size gets bigger. 

BTW, I tried setting the checksum type to NULL in conf and it failed the same way. 
                
> Allow users to specify a checksum type on create()
> --------------------------------------------------
>
>                 Key: HADOOP-8240
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8240
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>             Fix For: 0.23.3, 2.0.2-alpha
>
>         Attachments: hadoop-8240-branch-0.23-alone.patch.txt, hadoop-8240.patch, hadoop-8240-post-hadoop-8700-br2-trunk.patch.txt,
hadoop-8240-post-hadoop-8700-br2-trunk.patch.txt, hadoop-8240-trunk-branch2.patch.txt, hadoop-8240-trunk-branch2.patch.txt,
hadoop-8240-trunk-branch2.patch.txt
>
>
> Per discussion in HADOOP-8060, a way for users to specify a checksum type on create()
is needed. The way FileSystem cache works makes it impossible to use dfs.checksum.type to
achieve this. Also checksum-related API is at Filesystem-level, so we prefer something at
that level, not hdfs-specific one.  Current proposal is to use CreatFlag.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message