hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1285) ChecksumFileSystem : Can't read when io.file.buffer.size < bytePerChecksum
Date Sat, 21 Apr 2007 16:13:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490631
] 

Raghu Angadi commented on HADOOP-1285:
--------------------------------------

Btw, this test showed that ChecksumFS returned  53 bytes without checking CRC for full 512
bytes. 

> ChecksumFileSystem : Can't read when io.file.buffer.size < bytePerChecksum
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-1285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1285
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.12.3
>            Reporter: Raghu Angadi
>
> Looks like ChecksumFileSystem fails to read a file when bytesPerChecksum is larger than
io.file.buffer.size. Default for bytesPerChecksum  and buffer size are 512 and 4096, so default
config might not see the problem.
> I noticed this problem when I was testing block level CRCs with different configs.
> How to reproduce with latest trunk:
> Copy a text  file larger than 512 bytes to dfs : bin/hadoop fs -copyFromLocal ~/tmp/x.txt
x.txt
> then set io.file.buffer.size to something smaller than 512 (say 53). Now try to read
the file :
>  bin/hadoop dfs -cat x.txt
> This will print only the first 53 characters.
> The following code or comment at  ChecksumFileSystem.java:163 seems suspect. But not
sure if more changes are required:
> {code}
>     public int read(byte b[], int off, int len) throws IOException {
>       // make sure that it ends at a checksum boundary
>       long curPos = getPos();
>       long endPos = len+curPos/bytesPerSum*bytesPerSum;
>       return readBuffer(b, off, (int)(endPos-curPos));
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message