hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ZhuGuanyin (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1821) IFile.Reader should check whether data crc has checked before it stop reading.
Date Thu, 27 May 2010 06:31:37 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872118#action_12872118
] 

ZhuGuanyin commented on MAPREDUCE-1821:
---------------------------------------

positionToNextRecord() in IFile.Reader should check whether checksumIn had crc checked when
reader get EOF_MARKER for currentKeyLength and currentValueLength.

if IFileInputStream had crc checked in doRead() , it should set a class private variable flag,
and the Reader could check througth a IFileInputStream  public method to query this variable.

if IFile.Reader get 2 byrtes -1 for keylength and valuelength and crc not checked in checksumIn,
 it would throw a exception to fail this task.

> IFile.Reader should check whether data crc has checked before it stop reading.
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1821
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1821
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: ZhuGuanyin
>            Assignee: ZhuGuanyin
>
> Currently IFile data has crc checked in IFileInputStream (doRead method), 
> Normally the IFile would end with 2 bytes of -1, which means EOF_MARKER for keylength
and valuelength, and then with 4 bytes crc checksum;
> IFileInputStream  checksumIn would check crc before IFile.Reader get EOF_MARKER, 
> IFile.Reader would stop reading when positionToNextRecord() read keylength EOF_MARKER(-1),and
valuelength  EOF_MARKER(-1);
> But if something error happened(IFile corrupted), if the IFileReader read -1, -1 not
at end of the IFile, the data may not checked! 
> Then Reader thought it had got all data and close reader......the task may fake success
without any WARNing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message