From mapreduce-dev-return-1999-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Thu May 27 06:26:06 2010 Return-Path: Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: (qmail 34262 invoked from network); 27 May 2010 06:26:06 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 May 2010 06:26:06 -0000 Received: (qmail 79469 invoked by uid 500); 27 May 2010 06:26:06 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 79255 invoked by uid 500); 27 May 2010 06:26:03 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 79247 invoked by uid 99); 27 May 2010 06:26:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 06:26:02 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 06:26:00 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o4R6PcI8020558 for ; Thu, 27 May 2010 06:25:38 GMT Message-ID: <1774671.16991274941538124.JavaMail.jira@thor> Date: Thu, 27 May 2010 02:25:38 -0400 (EDT) From: "ZhuGuanyin (JIRA)" To: mapreduce-dev@hadoop.apache.org Subject: [jira] Created: (MAPREDUCE-1821) IFile.Reader should check whether data crc has checked before it stop reading. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org IFile.Reader should check whether data crc has checked before it stop reading. ------------------------------------------------------------------------------ Key: MAPREDUCE-1821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1821 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Reporter: ZhuGuanyin Assignee: ZhuGuanyin Currently IFile data has crc checked in IFileInputStream (doRead method), Normally the IFile would end with 2 bytes of -1, which means EOF_MARKER for keylength and valuelength, and then with 4 bytes crc checksum; IFileInputStream checksumIn would check crc before IFile.Reader get EOF_MARKER, IFile.Reader would stop reading when positionToNextRecord() read keylength EOF_MARKER(-1),and valuelength EOF_MARKER(-1); But if something error happened(IFile corrupted), if the IFileReader read -1, -1 not at end of the IFile, the data may not checked! Then Reader thought it had got all data and close reader......the task may fake success without any WARNing. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.