hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jthie...@ina.fr
Subject Re : Re: Strange bug split a table in two
Date Wed, 18 Feb 2009 18:33:15 GMT
Is it possible to request split or compaction during intensive write ?



----- Message d'origine -----
De: stack <stack@duboce.net>
Date: Mercredi, Février 18, 2009 6:38 pm
Objet: Re: Strange bug split a table in two

> Jérôme:
> 
> Which version of hbase?
> 
> Enable DEBUG.  See FAQ for how.  Have you read the getting started 
> where it
> suggests you up the file descriptors?  See also end of the 
> troubleshootingpage for hadoop config. needed for hbase.
> 
> How big are your tables?  How many rows/regions?
> 
> St.Ack
> 
> 
> On Wed, Feb 18, 2009 at 7:57 AM, Jérôme Thièvre INA 
> <jthievre@ina.fr> wrote:
> 
> > Hi,
> >
> >
> > During batch insertion of rows with java client in a table, I have
> > requested
> > a split of this table with the HBase web interface.
> > The insertion process started to slowdown, and I think it's 
> normal, but
> > then
> > it stopped with no exception.
> >
> > So I stopped the hbase cluster with bin/stop-hbase.sh and every 
> region> server stopped normally (I don't kill any process).
> >
> > I take a look at the logs :
> >
> > *master logs firest exceptions :
> >
> > *2009-02-18 15:48:27,969 INFO 
> org.apache.hadoop.hbase.master.ServerManager:> Received 
> MSG_REPORT_SPLIT: metadata_table,r:
> > http://net.series-tv.www/index.php?showforum=197,1234542589092:
> > metadata_table,r:
> > http://net.series-tv.www/index.php?showforum=197,1234542589092 
> split;> daughters: metadata_table,r:
> > http://net.series-tv.www/index.php?showforum=197,1234968484302,
> > metadata_table,r:
> > http://net.series-tv.www/index.php?showtopic=6973,1234968484302 from
> > 10.1.188.16:60020
> > 2009-02-18 15:48:27,969 INFO 
> org.apache.hadoop.hbase.master.RegionManager:> assigning region 
> metadata_table,r:> http://net.series-
> tv.www/index.php?showtopic=6973,1234968484302 to server
> > 10.1.188.16:60020
> > 2009-02-18 15:48:27,970 INFO 
> org.apache.hadoop.hbase.master.RegionManager:> assigning region 
> metadata_table,r:> http://net.series-
> tv.www/index.php?showforum=197,1234968484302 to server
> > 10.1.188.16:60020
> > 2009-02-18 15:48:29,555 INFO 
> org.apache.hadoop.hbase.master.ServerManager:> Received 
> MSG_REPORT_PROCESS_OPEN: metadata_table,r:
> >
> > http://fr.weborama.pro/fcgi-
> bin/comptage.fcgi?ID=175809&MEDIA=MAIL&PAGE=1&ZONE=50000,1234968501145from>
10.1.188.179:60020
> > 2009-02-18 15:48:29,555 INFO 
> org.apache.hadoop.hbase.master.ServerManager:> Received 
> MSG_REPORT_OPEN: metadata_table,r:
> >
> > 
> http://info.sirti.www/spip.php?id_article=320&page=galerie2,1234968501145from>
10.1.188.179:60020
> > 2009-02-18 15:48:29,555 INFO
> > org.apache.hadoop.hbase.master.ProcessRegionOpen$1: 
> metadata_table,r:>
> > 
> http://info.sirti.www/spip.php?id_article=320&page=galerie2,1234968501145open>
on
> > 10.1.188.179:60020
> > 2009-02-18 15:48:29,555 INFO
> > org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
> > metadata_table,r:
> > 
> http://info.sirti.www/spip.php?id_article=320&page=galerie2,1234968501145in> region
.META.,,1 with startcode 1234946982368 and server
> > 10.1.188.179:60020
> > 2009-02-18 15:48:30,994 INFO 
> org.apache.hadoop.hbase.master.ServerManager:> Received 
> MSG_REPORT_PROCESS_OPEN: metadata_table,r:
> > http://net.series-tv.www/index.php?showforum=197,1234968484302 from
> > 10.1.188.16:60020
> > 2009-02-18 15:48:30,995 INFO 
> org.apache.hadoop.hbase.master.ServerManager:> Received 
> MSG_REPORT_OPEN: metadata_table,r:
> > http://net.series-tv.www/index.php?showtopic=6973,1234968484302 from
> > 10.1.188.16:60020
> > 2009-02-18 15:48:30,995 INFO
> > org.apache.hadoop.hbase.master.ProcessRegionOpen$1: 
> metadata_table,r:> http://net.series-
> tv.www/index.php?showtopic=6973,1234968484302 open on
> > 10.1.188.16:60020
> > 2009-02-18 15:48:30,995 INFO
> > org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
> > metadata_table,r:
> > http://net.series-tv.www/index.php?showtopic=6973,1234968484302 
> in region
> > .META.,,1 with startcode 1234946972127 and server 10.1.188.16:60020
> > 2009-02-18 15:48:40,006 INFO 
> org.apache.hadoop.hbase.master.ServerManager:> Received 
> MSG_REPORT_CLOSE: metadata_table,r:
> > http://net.series-tv.www/index.php?showforum=197,1234968484302:
> > java.io.IOException: Could not obtain block: blk_-
> 6029004777792863005_53535>
> > 
> file=/hbase/metadata_table/1933533649/location/info/912096781946009771.309611126>
   at
> >
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)>
   at
> >
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)>
   at
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)>    at
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1593)>    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:320)
> >    at java.io.DataInputStream.readUTF(DataInputStream.java:572)
> >    at java.io.DataInputStream.readUTF(DataInputStream.java:547)
> >    at 
> org.apache.hadoop.hbase.io.Reference.readFields(Reference.java:105)>    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HStoreFile.readSplitInfo(HStoreFile.java:295)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:436)>    at
org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:230)
> >    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)>
   at
> > 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)>    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)>
   at java.lang.Thread.run(Thread.java:619)
> >  from 10.1.188.16:60020
> > 2009-02-18 15:48:42,681 INFO 
> org.apache.hadoop.hbase.master.RegionManager:> assigning region 
> metadata_table,r:> http://net.series-
> tv.www/index.php?showforum=197,1234968484302 to server
> > 10.1.188.149:60020
> > 2009-02-18 15:48:44,580 INFO 
> org.apache.hadoop.hbase.master.ServerManager:> Received 
> MSG_REPORT_CLOSE: metadata_table,r:
> >
> > http://fr.weborama.pro/fcgi-
> bin/comptage.fcgi?ID=175809&MEDIA=MAIL&PAGE=1&ZONE=50000,1234968501145>
:
> > java.io.IOException: Could not obtain block: 
> blk_1599510651183165167_53487>
> > 
> file=/hbase/metadata_table/1127743078/type/info/5407628626802748081.1381909621>  
 at
> >
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)>
   at
> >
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)>
   at
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)>    at
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1593)>    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:320)
> >    at java.io.DataInputStream.readUTF(DataInputStream.java:572)
> >    at java.io.DataInputStream.readUTF(DataInputStream.java:547)
> >    at 
> org.apache.hadoop.hbase.io.Reference.readFields(Reference.java:105)>    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HStoreFile.readSplitInfo(HStoreFile.java:295)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:436)>    at
org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:230)
> >    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)>
   at
> > 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)>    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)>
   at java.lang.Thread.run(Thread.java:619)
> >  from 10.1.188.179:60020
> > *
> > And after few exception on differents regions :
> >
> > *009-02-18 15:49:29,955 WARN 
> org.apache.hadoop.hbase.master.BaseScanner:> Scan one META region: 
> {regionname: .META.,,1, startKey: <>, server:
> > 10.1.188.16:60020}
> > java.io.IOException: java.io.IOException: HStoreScanner failed 
> construction>    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:88)>
   at
> > 
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2125)>    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1989)>
   at
> > 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1180)>    at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1700)>
   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> >    at
> >
> > 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)>
   at java.lang.reflect.Method.invoke(Method.java:597)
> >    at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)>    at
> > 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)> Caused
by: java.io.IOException: Could not obtain block:
> > blk_6746847995679537137_51100
> > file=/hbase/.META./1028785192/info/mapfiles/2067000542076825598/data
> >    at
> >
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)>
   at
> >
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)>
   at
> > 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)>    at java.io.DataInputStream.readFully(DataInputStream.java:178)
> >    at java.io.DataInputStream.readFully(DataInputStream.java:152)
> >    at
> > 
> org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)>    at
> >
> > 
> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)>
   at
> >
> > 
> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)>
   at
> >
> > 
> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)>
   at
> >
> > 
> org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)>
   at
> >
> > 
> org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)>
   at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
> >    at
> >
> > 
> org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)>
   at
> >
> > 
> org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)>  
 at
> >
> > 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)>
   at
> >
> > 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:67)>
   ... 10 more
> >
> >    at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> >    at
> >
> > 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)>
   at
> >
> > 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)>
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >    at
> >
> > 
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95)>
   at
> > 
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)>    at
> >
> > 
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)>
   at
> >
> > 
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)>
   at
> > 
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)>    at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
> >
> > When I restart the cluster I have two instances of my table (with 
> the same
> > name).
> >
> > I have just requested a major compaction, and everything seems to 
> be fine.
> > Hadoop fsck don't find any problems.
> >
> > I have some questions :
> >
> > Does the .META or .ROOT tables could have been corrupted, do you 
> think some
> > data have been lost from the table ?
> > Is it safe to split or compact table during writes ? I thought it 
> was ok.
> >
> > Jérôme Thièvre
> >
> 

Mime
View raw message