hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Yet another Hbase failure
Date Thu, 19 Mar 2009 17:52:54 GMT
Can you pull this file from your hdfs?

/hbase/.META./1028785192/info/mapfiles/3882153324238090640/index

Michael, these seem to be issues related to ulimit -n or to xceivers count
or to dfsclient timeout.  Can you confirm for us that you have indeed have
correct configurations in place.  Tell us also how you have these
configurations deployed.

Can you paste the hbase-site.xml, the hadoop-site.xml, and the first few
lines of your log when it starts up where it shows the jvm and ulimit
settings?

Thanks,
St.Ack


On Thu, Mar 19, 2009 at 10:00 AM, Michael Dagaev
<michael.dagaev@gmail.com>wrote:

> Yes, unfortunately  it is semi-production data.
> I will try to get on IRC.
>
> I am afraid though it will be difficult to fix the data
> since I won't be able to start and start HDFS/Hbase.
>
> I changed "dfs.data.dir" to point out to another file system and
> HDFS/Hbase are running now.
>
>  I switched the dfs.data.dir to another file system, so Hadoop and
> Hbase are not working now with the   Can we fix the meta region while
> Hadoop and Hbase are working with another
>
> On Thu, Mar 19, 2009 at 2:59 PM, Jean-Daniel Cryans <jdcryans@apache.org>
> wrote:
> > Michael,
> >
> > This may be related to your other problem. We need to find the root of
> > what seems to be data lost which is probable.
> >
> > Is it production data? If so, we may have to fix your meta region. Any
> > chance you drop on IRC?
> >
> > J-D
> >
> > On Thu, Mar 19, 2009 at 7:06 AM, Michael Dagaev
> > <michael.dagaev@gmail.com> wrote:
> >> Hi, all
> >>
> >>    We are running a small hbase 0.18 cluster. It stopped working now.
> >> The master log contains "java.io.IOException: HStoreScanner failed
> >> construction" errors.
> >> Restart (stop/start-hbase) does not help. Hadoop fsck says that
> >> everything is fine.
> >>
> >> Has anybody run into such a problem ?
> >>
> >> Thank you for your cooperation,
> >> M.
> >>
> >> 2009-03-19 10:57:01,306 WARN org.apache.hadoop.hbase.master.BaseScanner:
> >> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
> >> 10.251.142.47:60020}
> >> java.io.IOException: HStoreScanner failed construction
> >>        at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
> >>        at
> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:70)
> >>        at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1916)
> >>        at
> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1954)
> >>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1345)
> >>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1175)
> >>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>        at java.lang.reflect.Method.invoke(Method.java:597)
> >>        at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> >>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> >> Caused by: java.io.IOException: Could not obtain block:
> >> blk_-2893342489105927361_2846854
> >> file=/hbase/.META./1028785192/info/mapfiles/3882153324238090640/index
> >>        at
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462)
> >>        at
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312)
> >>        at
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417)
> >>        at java.io.DataInputStream.readFully(DataInputStream.java:178)
> >>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
> >>        at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1453)
> >>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> >>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1420)
> >>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1415)
> >>        at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
> >>        at
> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:632)
> >>        at
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:714)
> >>        at
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:413)
> >>        at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> >>        at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:67)
> >>        ... 11 more
> >>
> >>        at sun.reflect.GeneratedConstructorAccessor11.newInstance(Unknown
> Source)
> >>        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >>        at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >>        at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> >>        at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:191)
> >>        at
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:74)
> >>        at
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
> >>        at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:139)
> >>        at org.apache.hadoop.hbase.Chore.run(Chore.java:62)
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message