hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ZhuGuanyin (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1821) IFile.Reader should check whether data crc has checked before it stop reading.
Date Thu, 27 May 2010 06:31:37 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872118#action_12872118

ZhuGuanyin commented on MAPREDUCE-1821:

positionToNextRecord() in IFile.Reader should check whether checksumIn had crc checked when
reader get EOF_MARKER for currentKeyLength and currentValueLength.

if IFileInputStream had crc checked in doRead() , it should set a class private variable flag,
and the Reader could check througth a IFileInputStream  public method to query this variable.

if IFile.Reader get 2 byrtes -1 for keylength and valuelength and crc not checked in checksumIn,
 it would throw a exception to fail this task.

> IFile.Reader should check whether data crc has checked before it stop reading.
> ------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1821
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1821
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: ZhuGuanyin
>            Assignee: ZhuGuanyin
> Currently IFile data has crc checked in IFileInputStream (doRead method), 
> Normally the IFile would end with 2 bytes of -1, which means EOF_MARKER for keylength
and valuelength, and then with 4 bytes crc checksum;
> IFileInputStream  checksumIn would check crc before IFile.Reader get EOF_MARKER, 
> IFile.Reader would stop reading when positionToNextRecord() read keylength EOF_MARKER(-1),and
valuelength  EOF_MARKER(-1);
> But if something error happened(IFile corrupted), if the IFileReader read -1, -1 not
at end of the IFile, the data may not checked! 
> Then Reader thought it had got all data and close reader......the task may fake success
without any WARNing.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message