hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik Manimaran <karthikmanima...@gmail.com>
Subject Re: HBase startup issues
Date Mon, 07 Feb 2011 22:04:05 GMT
Thanks for the hint J-D. The filesystem got filled up causing this 
issue. All clear now.

regards,
Karthik Manimaran.

On 02/07/2011 01:00 PM, Jean-Daniel Cryans wrote:
> Your HDFS doesn't look healthy to me, it doesn't accept new blocks!
> Check the datanode / namenode logs and see if any exception in there.
>
> J-D
>
> On Mon, Feb 7, 2011 at 12:58 PM, Karthik Manimaran
> <karthikmanimaran@gmail.com>  wrote:
>    
>> Hi,
>>
>> My HBase setup was running fine for a couple of months and all of a sudden
>> the following issues has cropped up. The master will shutdown immediately
>> after startup. The Hadoop datanode is running fine and hdfs status is
>> Healthy. Any ideas on what could be happening and steps on how to fix/bypass
>> these errors and get HBase running again? Here are the relevant errors from
>> logs:
>>
>> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /hbase/.META./1028785192/info/89629418893503418 could only be
>> replicated to 0 nodes, instead of 1
>>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>>         at org.apache.hadoop.ipc.Client.call(Client.java:739)
>>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>>         at $Proxy0.addBlock(Unknown Source)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>         at $Proxy0.addBlock(Unknown Source)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>>
>> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Could
>> not get block locations. Source file
>> "/hbase/.META./1028785192/info/89629418893503418" - Aborting...
>> 2011-02-07 14:39:45,644 FATAL
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
>> required. Forcing server shutdown
>> org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1
>>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946)
>>         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839)
>>         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
>>         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
>> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>> File /hbase/.META./1028785192/info/89629418893503418 could only be
>> replicated to 0 nodes, instead of 1
>>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>>         at org.apache.hadoop.ipc.Client.call(Client.java:739)
>>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>>         at $Proxy0.addBlock(Unknown Source)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>         at $Proxy0.addBlock(Unknown Source)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>> 2011-02-07 14:39:45,662 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>> request=0.0, regions=73, stores=227, storefiles=184,
>> storefileIndexSize=1, memstoreSize=0, usedHeap=37, maxHeap=1993,
>> blockCacheSize=1962504, blockCacheFree=416130968, blockCacheCount=1,
>> blockCacheHitRatio=75
>> 2011-02-07 14:39:45,662 INFO
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
>> RegionServer:0.cacheFlusher exiting
>> 2011-02-07 14:39:48,481 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping server on 37024
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 3 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping IPC Server listener on 37024
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 0 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 1 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 2 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 5 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 7 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 9 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 4 on 37024: exiting
>> 2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 6 on 37024: exiting
>> 2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 8 on 37024: exiting
>> 2011-02-07 14:39:48,488 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping IPC Server Responder
>> 2011-02-07 14:39:48,488 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping
>> infoServer
>> 2011-02-07 14:39:48,490 DEBUG
>> org.apache.hadoop.hbase.regionserver.HLog: closing hlog writer in
>> hdfs://ec2-75-101-139-144.compute-1.amazonaws.com:9000/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.LogFlusher:
>> RegionServer:0.logFlusher exiting
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
>> RegionServer:0.compactor exiting
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
>> RegionServer:0.majorCompactionChecker exiting
>> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
>> could only be replicated to 0 nodes, instead of 1
>>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>>         at org.apache.hadoop.ipc.Client.call(Client.java:739)
>>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>>         at $Proxy0.addBlock(Unknown Source)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>         at $Proxy0.addBlock(Unknown Source)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
>>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>>
>> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Could
>> not get block locations. Source file
>> "/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949"
>> - Aborting...
>> 2011-02-07 14:39:48,501 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close
>> log in abort
>> java.io.IOException: java.io.IOException: File
>> /hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
>> could only be replicated to 0 nodes, instead of 1
>>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>         at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>         at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>         at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
>>         at java.lang.Thread.run(Thread.java:619)
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
>> user_activity,,1288475114682
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Closing
>> user_activity,,1288475114682: compactions&  flushes disabled
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
>> region, no outstanding scanners on user_activity,,1288475114682
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
>> outstanding on region user_activity,,1288475114682
>> 2011-02-07 14:39:48,503 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed comments
>> 2011-02-07 14:39:48,503 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed company_ratings
>> 2011-02-07 14:39:48,504 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed details
>> 2011-02-07 14:39:48,504 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed event
>> 2011-02-07 14:39:48,505 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed event_details
>> 2011-02-07 14:39:48,505 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed likes
>> 2011-02-07 14:39:48,505 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed tags
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed user_ratings
>> 2011-02-07 14:39:48,506 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Closed
>> user_activity,,1288475114682
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
>> company_analytics_post_a_job_tab,,1288475078918
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Closing
>> company_analytics_post_a_job_tab,,1288475078918: compactions&  flushes
>> disabled
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
>> region, no outstanding scanners on
>> company_analytics_post_a_job_tab,,1288475078918
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
>> outstanding on region company_analytics_post_a_job_tab,,1288475078918
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed system_details
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed viewer_details
>>
>>
>> Thanks,
>> Karthik Manimaran.
>>
>>      


Mime
View raw message