Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sun, 30 Sep 2012 07:42:08 +1100 (NCT)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <43749341.143652.1348951328769.JavaMail.jiratomcat@arcas>
In-Reply-To: <783540961.116097.1348480807901.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error in
 HFile V2 due to incorrect split into intermediate index blocks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HBASE-6871?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1346=
6303#comment-13466303 ]=20

stack commented on HBASE-6871:
------------------------------

[~lhofhansl] Looks like its in there if I do svn diff -r1391284:1391742.  H=
mm... yeah, my fault, 0.94 has categories.  Let me put it back.
               =20
> HFileBlockIndex Write Error in HFile V2 due to incorrect split into inter=
mediate index blocks
> -------------------------------------------------------------------------=
--------------------
>
>                 Key: HBASE-6871
>                 URL: https://issues.apache.org/jira/browse/HBASE-6871
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile
>    Affects Versions: 0.94.1
>         Environment: redhat 5u4
>            Reporter: Fenng Wang
>            Assignee: Mikhail Bautin
>            Priority: Critical
>             Fix For: 0.92.3, 0.94.3, 0.96.0
>
>         Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 6871.094.add=
endum.txt, 6871-0.94.txt, 6871-0.94v2.txt, 6871-hfile-index-0.92.txt, 6871-=
hfile-index-0.92-v2.txt, 6871.txt, 6871v2.txt, 787179746cc347ce9bb36f1989d1=
7419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f5=
9dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch=
, D5703.5.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_=
index.sh
>
>
> After writing some data, compaction and scan operation both failure, the =
exception message is below:
> 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compac=
tions.CompactionRequest: Compaction failed regionName=3Dhfile_test,,1347778=
722498.d220df43fb9d8af4633bd7f547613f9e., storeName=3Dpage_info, fileCount=
=3D7, fileSize=3D1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.=
3k), priority=3D9, time=3D45826250816757428java.io.IOException: Could not r=
eseek StoreFileScanner[HFileScanner for reader reader=3Dhdfs://hadoopdev1.c=
m6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118=
f58de47ad9d87cac438ee0895, compression=3Dlzo, cacheConf=3DCacheConfig:enabl=
ed [cacheDataOnRead=3Dtrue] [cacheDataOnWrite=3Dfalse] [cacheIndexesOnWrite=
=3Dfalse] [cacheBloomsOnWrite=3Dfalse] [cacheEvictOnClose=3Dfalse] [cacheCo=
mpressed=3Dfalse], firstKey=3Dhttp://com.truereligionbrandjeans.www/Womens_=
Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Women=
s_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Wome=
ns_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Wom=
ens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Wo=
mens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweater=
s/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_=
Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Wome=
ns_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c=
/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Swea=
ters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449=
/DeleteColumn, lastKey=3Dhttp://com.trura.www//page_info:page_type/13477633=
95089/Put, avgKeyLen=3D776, avgValueLen=3D4, entries=3D12853, length=3D2286=
11, cur=3Dhttp://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl=
/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=3D1/ts=3D0] to key=
 http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.h=
tml/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=3D0/ts=3D0
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(S=
toreFileScanner.java:178)       =20
>         at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.do=
RealSeek(NonLazyKeyValueScanner.java:54)       =20
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedS=
eek(KeyValueHeap.java:299)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyVa=
lueHeap.java:244)       =20
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(Store=
Scanner.java:521)       =20
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreSc=
anner.java:402)
>         at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.=
java:1570)       =20
>         at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:=
997)       =20
>         at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.j=
ava:1216)
>         at org.apache.hadoop.hbase.regionserver.compactions.CompactionReq=
uest.run(CompactionRequest.java:250)       =20
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP=
oolExecutor.java:886)       =20
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got I=
NTERMEDIATE_INDEX: blockType=3DINTERMEDIATE_INDEX, onDiskSizeWithoutHeader=
=3D8514, uncompressedSizeWithoutHeader=3D131837, prevBlockOffset=3D-1, data=
BeginsWith=3D\x00\x00\x00\x9B\x00\x00\x00\x00\x00\x00\x03#\x00\x00\x050\x00=
\x00\x08\xB7\x00\x00\x0Cr\x00\x00\x0F\xFA\x00\x00\x120, fileOffset=3D218942=
        at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.validateBlockType=
(HFileReaderV2.java:378)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFile=
ReaderV2.java:331)        at org.apache.hadoop.hbase.io.hfile.HFileBlockInd=
ex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScanner=
V2.seekTo(HFileReaderV2.java:455)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScanner=
V2.reseekTo(HFileReaderV2.java:493)       =20
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAt=
OrAfter(StoreFileScanner.java:242)       =20
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(S=
toreFileScanner.java:167)
> After some debug works=EF=BC=8CI found that when hfile closing, if the ro=
otChunk is empty, the only one curInlineChunk will upgrade to root chunk. B=
ut if the last block flushing make curInlineChunk exceed max index block si=
ze, the root chunk(upgrade from curInlineChunk) will be splited into interm=
ediate index blocks, and the index level is set to 2. So when BlockIndexRea=
der read the root index, it expects the next level index block is leaf inde=
x(index level=3D2), but the on disk index block is intermediate block, the =
error happened.=20
> After I add some code to check curInlineChunk's size when rootChunk is em=
pty in shouldWriteBlock(boolean closing), this bug can be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira