Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B821A10B41 for ; Thu, 16 Jan 2014 18:35:21 +0000 (UTC) Received: (qmail 82402 invoked by uid 500); 16 Jan 2014 18:35:20 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 82341 invoked by uid 500); 16 Jan 2014 18:35:19 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 82331 invoked by uid 99); 16 Jan 2014 18:35:19 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jan 2014 18:35:19 +0000 Date: Thu, 16 Jan 2014 18:35:19 +0000 (UTC) From: "Kousuke Saruta (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-5761) DataNode fails to validate integrity for checksum type NULL when DataNode recovers MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873719#comment-13873719 ] Kousuke Saruta commented on HDFS-5761: -------------------------------------- Thanks for your comment, Uma. At first, I thought same as you. I thought it's good to branch the logic depending on whether checksum type is NULL or not. But, on second thought, BlockPoolSlice should not have logic which depends specific checksum algorithm. How to verify is responsibility of each checksum algorithm. > DataNode fails to validate integrity for checksum type NULL when DataNode recovers > ----------------------------------------------------------------------------------- > > Key: HDFS-5761 > URL: https://issues.apache.org/jira/browse/HDFS-5761 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Affects Versions: 3.0.0 > Reporter: Kousuke Saruta > Assignee: Kousuke Saruta > Attachments: HDFS-5761.patch > > > When DataNode is down during writing blocks, the blocks are not filinalized and the next time DataNode recovers, integrity validation will run. > But if we use NULL for checksum algorithm (we can set NULL to dfs.checksum.type), DataNode will fail to validate integrity and cannot be up. > The cause is in BlockPoolSlice#validateIntegrity. > In the method, there is following code. > {code} > long numChunks = Math.min( > (blockFileLen + bytesPerChecksum - 1)/bytesPerChecksum, > (metaFileLen - crcHeaderLen)/checksumSize); > {code} > When we choose NULL checksum, checksumSize is 0 so ArithmeticException will be thrown and DataNode cannot be up. -- This message was sent by Atlassian JIRA (v6.1.5#6160)