hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1285) ChecksumFileSystem : Can't read when io.file.buffer.size < bytePerChecksum
Date Sat, 21 Apr 2007 00:40:15 GMT
ChecksumFileSystem : Can't read when io.file.buffer.size < bytePerChecksum

                 Key: HADOOP-1285
                 URL: https://issues.apache.org/jira/browse/HADOOP-1285
             Project: Hadoop
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.12.3
            Reporter: Raghu Angadi

Looks like ChecksumFileSystem fails to read a file when bytesPerChecksum is larger than io.file.buffer.size.
Default for bytesPerChecksum  and buffer size are 512 and 4096, so default config might not
see the problem.

I noticed this problem when I was testing block level CRCs with different configs.

How to reproduce with latest trunk:
Copy a text  file larger than 512 bytes to dfs : bin/hadoop fs -copyFromLocal ~/tmp/x.txt
then set io.file.buffer.size to something smaller than 512 (say 53). Now try to read the file

 bin/hadoop dfs -cat x.txt

This will print only the first 53 characters.

The following code or comment at  ChecksumFileSystem.java:163 seems suspect. But not sure
if more changes are required:
    public int read(byte b[], int off, int len) throws IOException {
      // make sure that it ends at a checksum boundary
      long curPos = getPos();
      long endPos = len+curPos/bytesPerSum*bytesPerSum;
      return readBuffer(b, off, (int)(endPos-curPos));

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message