Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 4B1E8200AEE for ; Wed, 4 May 2016 00:57:15 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 49FD31609F5; Wed, 4 May 2016 00:57:15 +0200 (CEST) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A654F1609F8 for ; Wed, 4 May 2016 00:57:14 +0200 (CEST) Received: (qmail 8885 invoked by uid 500); 3 May 2016 22:57:13 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 8862 invoked by uid 99); 3 May 2016 22:57:13 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2016 22:57:13 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 4EF4F2C1F6A for ; Tue, 3 May 2016 22:57:13 +0000 (UTC) Date: Tue, 3 May 2016 22:57:13 +0000 (UTC) From: "Appy (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-11625) Reading datablock throws "Invalid HFile block magic" and can not switch to hdfs checksum MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 03 May 2016 22:57:15 -0000 [ https://issues.apache.org/jira/browse/HBASE-11625?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D152= 69794#comment-15269794 ]=20 Appy commented on HBASE-11625: ------------------------------ Here's how to reproduce. I have uploaded two files: - correct-hfile : Generated using hbase shell to write two rows to a new ta= ble and flushing it. - corrupted-header-hfile: Changed one character in 'correct-hfile' belongin= g to block magic. Use {{cmp -b correct-hfile corrupted-header-hfile}} to se= e the diff. Since I messed with the file, it should report that data was corrupted but = instead it throws 'Invalid hfile block magic' exception (see below output). The fix, as mentioned above, is to first do checksum and then make HFileBlo= ck instance. {noformat} appy@asharma-MBP ~/apache/hbase/bin (master) =E2=86=92 ./hbase org.apache.= hadoop.hbase.io.hfile.HFile -p -f file:///Users/appy/apache/hbase/correct-h= file 2016-05-03 15:47:37,332 WARN [main] util.NativeCodeLoader: Unable to load = native-hadoop library for your platform... using builtin-java classes where= applicable 2016-05-03 15:47:37,534 INFO [main] hfile.CacheConfig: CacheConfig:disable= d K: 1/cf:1/1462311794785/Put/vlen=3D5/seqid=3D4 V: 11111 K: 1/cf:2/1462311803576/Put/vlen=3D6/seqid=3D5 V: 222222 Scanned kv count -> 2 appy@asharma-MBP ~/apache/hbase/bin (master) =E2=86=92 ./hbase org.apache.= hadoop.hbase.io.hfile.HFile -p -f file:///Users/appy/apache/hbase/corrupted= -header-hfile 2016-05-03 15:47:43,932 WARN [main] util.NativeCodeLoader: Unable to load = native-hadoop library for your platform... using builtin-java classes where= applicable 2016-05-03 15:47:44,089 INFO [main] hfile.CacheConfig: CacheConfig:disable= d 2016-05-03 15:47:44,301 ERROR [main] hfile.HFilePrettyPrinter: Error readin= g file:/Users/appy/apache/hbase/corrupted-header-hfile java.io.IOException: Invalid HFile block magic: DcTABLK* =09at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154) =09at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167) =09at org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:25= 5) =09at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDat= aInternal(HFileBlock.java:1684) =09at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDat= a(HFileBlock.java:1507) =09at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV= 2.java:446) =09at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFile= ReaderV2.java:914) =09at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFile= PrettyPrinter.java:269) =09at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPr= inter.java:230) =09at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) =09at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyP= rinter.java:552) =09at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:879) {noformat} > Reading datablock throws "Invalid HFile block magic" and can not switch t= o hdfs checksum=20 > -------------------------------------------------------------------------= ---------------- > > Key: HBASE-11625 > URL: https://issues.apache.org/jira/browse/HBASE-11625 > Project: HBase > Issue Type: Bug > Components: HFile > Affects Versions: 0.94.21, 0.98.4, 0.98.5, 1.0.1.1, 1.0.3 > Reporter: qian wang > Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: 2711de1fdf73419d9f8afc6a8b86ce64.gz, HBASE-11625.pat= ch, correct-hfile, corrupted-header-hfile > > > when using hbase checksum,call readBlockDataInternal() in hfileblock.java= , it could happen file corruption but it only can switch to hdfs checksum i= nputstream till validateBlockChecksum(). If the datablock's header corrupte= d when b =3D new HFileBlock(),it throws the exception "Invalid HFile block = magic" and the rpc call fail -- This message was sent by Atlassian JIRA (v6.3.4#6332)