Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5330E10FA9 for ; Wed, 15 Jan 2014 05:06:55 +0000 (UTC) Received: (qmail 66966 invoked by uid 500); 15 Jan 2014 05:06:51 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 66499 invoked by uid 500); 15 Jan 2014 05:06:49 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 66485 invoked by uid 99); 15 Jan 2014 05:06:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jan 2014 05:06:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [98.138.229.29] (HELO nm36.bullet.mail.ne1.yahoo.com) (98.138.229.29) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jan 2014 05:06:40 +0000 Received: from [127.0.0.1] by nm36.bullet.mail.ne1.yahoo.com with NNFMP; 15 Jan 2014 05:06:18 -0000 Received: from [98.138.101.132] by nm36.bullet.mail.ne1.yahoo.com with NNFMP; 15 Jan 2014 05:03:18 -0000 Received: from [98.139.212.153] by tm20.bullet.mail.ne1.yahoo.com with NNFMP; 15 Jan 2014 05:03:18 -0000 Received: from [98.139.212.204] by tm10.bullet.mail.bf1.yahoo.com with NNFMP; 15 Jan 2014 05:03:18 -0000 Received: from [127.0.0.1] by omp1013.mail.bf1.yahoo.com with NNFMP; 15 Jan 2014 05:03:18 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 504515.93847.bm@omp1013.mail.bf1.yahoo.com Received: (qmail 97510 invoked by uid 60001); 15 Jan 2014 05:03:18 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1389762198; bh=wDAyihUeYT4m+jr8rI8EWe19DDdh6pnCgMKIUBOoU8w=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=4/pIPU5k1vYON9yTt0tKs1CvqwcJRidRU1BZvkIwUOJSUBv/A7RL1bHPHLse8j/45MP9OttPThIO4LzxznT2LE+KOUGb1DreAPAZbwjjeGB2Wm54a5RHGiUPpphZ6d7NQo6a2Cz8P8q/t79lsoKbcY+kiNnXtgWkXJGNVCuwciw= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=qlQ3Ye6Cj5ujJvjBAz+l8sACW1UBadyFzLmBOM+97qAk1vyk58lKJMIGsT7g2Q6Qyct73xbnjr8lCuL5oGoU3Hjyx25nL8KYAEqMffGA1TLho7exZrUSwRcZPzORY+VUccrwwf5Y2CyYtS4lclz/GGhQxfmuIH4ZixhGouowT0w=; X-YMail-OSG: k2HtKX8VM1kZXhdHltexH0Y3LB3lH3DGHaUqrXXQcOlcWuC WIqKRXg30VENhNf2HzmokA.DcVOhtYpNYYC5wky.DiG9A1RnL_5Z3vrMWOgy fa_WrCObNk8XOluKbSDBa_I_6HGdJmNaryG6GmwAAxBvGYDtDCpmrLzuIa_Q SXDoh0n9EXOKfZFXbzfC4QbFTX4ZJ30uJsvBYu3Tq8DUJPNb7dYwbvUgSx1r YBfy1cSOBI.QRT_O17r9WY.kPGRLU0EYVEbopLZwXnAaZw4Nj.27IoE9C392 .t9g3U8B9JF1rnVYIf7LykOIdDylMwdw.r2KLeDfU9PeozMEUxn4V7uh5o7I V.lOAPG_k7AgCWLLAtSIwNssRx.LpmNNezfmkiZqTWJawhEg4Y_MfO5kpFGL E1wUgv1D4vOkdZ.uAJCq41YgNh9RPNrRhKbCQfLHsczbQT_j_zZFv2NINrUi C7z.vHMvogGU0fbHu.9Ah_LpZraVXSu2k5ZyPkwOBA2lUpY9wH_VsGNNA9tl 6YFtAEMAIFUkDiTjTc5BfNK9B.gXjV9fb8wmU5o0yAwftHr4oFsamYjx0uzU dHHC8Nh1WVBl1dkm9b6WkqEKw..brdnI._J0KMvzPaQT44R5VoMnpXNLZZOz Bd_gIuMN117uCm0yCVA7u5g6SwsbIEUKY3juRBdRj6MdbYR5yJknVIGHxLlR NxJahQvp70Uqyp_B_DDaHZNGevzHTa8N2wmmE2kE.I0FUKi34_kujxAu0NpE - Received: from [24.4.148.188] by web140604.mail.bf1.yahoo.com via HTTP; Tue, 14 Jan 2014 21:03:18 PST X-Rocket-MIMEInfo: 002.001,V2hpY2ggdmVyc2lvbiBvZiBIYWRvb3A_CklmIHlvdSBnZXQgYSBkYXRhIGNlbnRlciB3aWRlIHBvd2VyIG91dGFnZSB5b3UgY2FuIGxvc2UgZGF0YS4KCkluIEhhZG9vcCAxLjEuMSBvciBsYXRlciB5b3UgZm9yY2UgYSBzeW5jIG9uIGJsb2NrIGNsb3NlLCBhbmQgdGh1cyB5b3Ugd29uJ3QgYXQgbGVhc3QgbG9zZSBhbnkgb2xkIGRhdGEgKGkuZS4gSEZpbGVzIHRoYXQgd2VyZSByZWNlbnRseSB3cml0dGVuIGR1ZSB0byBjb21wYWN0aW9ucykuCgpJIGhhdmUgYmxvZ2dlZCBhYm91dCB0aGF0IGhlcmU6CgpodHQBMAEBAQE- X-RocketYMMF: lhofhansl X-Mailer: YahooMailWebService/0.8.172.614 References: Message-ID: <1389762198.40530.YahooMailNeo@web140604.mail.bf1.yahoo.com> Date: Tue, 14 Jan 2014 21:03:18 -0800 (PST) From: lars hofhansl Reply-To: lars hofhansl Subject: Re: Corrupt HFile To: "user@hbase.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="1023258982-1428885504-1389762198=:40530" X-Virus-Checked: Checked by ClamAV on apache.org --1023258982-1428885504-1389762198=:40530 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Which version of Hadoop?=0AIf you get a data center wide power outage you c= an lose data.=0A=0AIn Hadoop 1.1.1 or later you force a sync on block close= , and thus you won't at least lose any old data (i.e. HFiles that were rece= ntly written due to compactions).=0A=0AI have blogged about that here:=0A= =0Ahttp://hadoop-hbase.blogspot.com/2013/07/protected-hbase-against-data-ce= nter.html=0A=0A=0A-- Lars=0A=0A=0A=0A________________________________=0A Fr= om: =E5=AE=BE=E8=8E=89=E9=87=91 =0ATo: user@hbase.apach= e.org =0ASent: Tuesday, January 14, 2014 7:36 PM=0ASubject: Corrupt HFile= =0A =0A=0AWe use hbase-0.94.10 and encounter corrupt hfile.=0A=0A=0A2014-01= -11 23:24:16,547 DEBUG org.apache.hadoop.hbase.util.FSUtils:=0ACreating=0Af= ile=3Dhdfs://dump002002.cm6:9000/hbase-0.90/cbu2/735414b148ed70e79f4c040696= 3bb0c9/.tmp/8a4869aafeae43ee8294bf7b65b92e63=0Awith permission=3Drwxrwxrwx= =0A2014-01-11 23:24:16,550 INFO=0Aorg.apache.hadoop.hbase.regionserver.Stor= eFile: Bloom filter type for=0Ahdfs://dump002002.cm6:9000/hbase-0.90/cbu2/7= 35414b148ed70e79f4c0406963bb0c9/.tmp/8a4869aafeae43ee8294bf7b65b92e63:=0ARO= W, CompoundBloomFilterWriter=0A2014-01-11 23:24:16,550 INFO=0Aorg.apache.ha= doop.hbase.regionserver.StoreFile: Delete Family Bloom filter=0Atype for=0A= hdfs://dump002002.cm6:9000/hbase-0.90/cbu2/735414b148ed70e79f4c0406963bb0c9= /.tmp/8a4869aafeae43ee8294bf7b65b92e63:=0ACompoundBloomFilterWriter=0A=0A20= 14-01-11 23:25:29,769 INFO org.apache.hadoop.hbase.regionserver.Store:=0ARe= naming compacted file at=0Ahdfs://dump002002.cm6:9000/hbase-0.90/cbu2/73541= 4b148ed70e79f4c0406963bb0c9/.tmp/8a4869aafeae43ee8294bf7b65b92e63=0Ato=0Ahd= fs://dump002002.cm6:9000/hbase-0.90/cbu2/735414b148ed70e79f4c0406963bb0c9/c= f/8a4869aafeae43ee8294bf7b65b92e63=0A=0A2014-01-11 23:25:29,914 INFO org.ap= ache.hadoop.hbase.regionserver.Store:=0ACompleted major compaction of 4 fil= e(s) in cf of=0Acbu2,/0614,1386589566547.735414b148ed70e79f4c0406963bb0c9. = into=0A8a4869aafeae43ee8294bf7b65b92e63, size=3D883.6 M; total size for sto= re is=0A884.5 M=0A=0A=0A2014-01-15 00:15:54,412 WARN org.apache.hadoop.hbas= e.io.hfile.HFile: File=0Ahdfs://dump002002.cm6:9000/hbase-0.90/cbu2/735414b= 148ed70e79f4c0406963bb0c9/cf/8a4869aafeae43ee8294bf7b65b92e63=0AStored chec= ksum value of -1392402101 at offset 64905 does not match=0Acomputed checksu= m 242524898, total data size 64950 Checksum data range=0Aoffset 16384 len 1= 6384 Header dump: magic: 4918304907327195946 blockType=0ADATA compressedBlo= ckSizeNoHeader 64884 uncompressedBlockSizeNoHeader 262185=0AprevBlockOffset= 71067534 checksumType CRC32 bytesPerChecksum 16384=0AonDiskDataSizeWithHea= der 64901=0A2014-01-15 00:15:54,412 WARN org.apache.hadoop.hbase.io.hfile.H= File: HBase=0Achecksum verification failed for file=0Ahdfs://dump002002.cm6= :9000/hbase-0.90/cbu2/735414b148ed70e79f4c0406963bb0c9/cf/8a4869aafeae43ee8= 294bf7b65b92e63=0Aat offset 71128929 filesize 926536062. Retrying read with= HDFS checksums=0Aturned on...=0A=0A2014-01-15 00:15:54,413 ERROR=0Aorg.apa= che.hadoop.hbase.regionserver.HRegionServer:=0Ajava.lang.ArrayIndexOutOfBou= ndsException=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Acom.hadoop.compression.lzo.= LzoDecompressor.setInput(LzoDecompressor.java:208)=0A=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at=0Aorg.apache.hadoop.io.compress.BlockDecompressorStream.decompres= s(BlockDecompressorStream.java:97)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.a= pache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:83= )=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.io.BufferedInputStream.fill(Buffere= dInputStream.java:218)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.io.BufferedInp= utStream.read1(BufferedInputStream.java:258)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 = at java.io.BufferedInputStream.read(BufferedInputStream.java:317)=0A=C2=A0 = =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java= :192)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.io.hfile.H= FileBlock$AbstractFSReader.decompress(HFileBlock.java:1461)=0A=C2=A0 =C2=A0= =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.= readBlockDataInternal(HFileBlock.java:1891)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 a= t=0Aorg.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HF= ileBlock.java:1734)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hb= ase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:342)=0A=C2=A0 =C2= =A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.io.hfile.HFileReaderV2$Abstr= actScannerV2.readNextDataBlock(HFileReaderV2.java:597)=0A=C2=A0 =C2=A0 =C2= =A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.ne= xt(HFileReaderV2.java:695)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.ha= doop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.j= ava:248)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regions= erver.StoreFileScanner.reseek(StoreFileScanner.java:161)=0A=C2=A0 =C2=A0 = =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regionserver.NonLazyKeyValueScan= ner.doRealSeek(NonLazyKeyValueScanner.java:54)=0A=C2=A0 =C2=A0 =C2=A0 =C2= =A0 at=0Aorg.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(= KeyValueHeap.java:321)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop= .hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:265)=0A=C2=A0 =C2= =A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regionserver.StoreScanner.re= seek(StoreScanner.java:545)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.h= adoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:411)=0A=C2=A0 = =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regionserver.KeyValueHeap= .next(KeyValueHeap.java:143)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.= hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.= java:3973)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regio= nserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4045)=0A=C2=A0 = =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regionserver.HRegion$Regi= onScannerImpl.nextRaw(HRegion.java:3916)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at= =0Aorg.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HReg= ion.java:3897)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.r= egionserver.HRegion$RegionScannerImpl.next(HRegion.java:3940)=0A=C2=A0 =C2= =A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regionserver.HRegion.get(HRe= gion.java:4867)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.= regionserver.HRegion.get(HRegion.java:4840)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 a= t=0Aorg.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja= va:2253)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop.hbase.regions= erver.HRegionServer.multi(HRegionServer.java:3828)=0A=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)=0A= =C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Asun.reflect.DelegatingMethodAccessorImpl.i= nvoke(DelegatingMethodAccessorImpl.java:25)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 a= t java.lang.reflect.Method.invoke(Method.java:597)=0A=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at=0Aorg.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writa= bleRpcEngine.java:320)=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=0Aorg.apache.hadoop= .hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)=0A=0A-- =0A*Best = Regards,*=0Alijin bin --1023258982-1428885504-1389762198=:40530--