hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GuoWei <wei....@wbkit.com>
Subject All the base region server going down
Date Tue, 26 Nov 2013 03:27:03 GMT
Dear,

Please help me to find out why all region servers going down at 2013-11-25 16:20. 



The logs list below  are logs from master and one slave. 


From Master:

2013-11-25 18:06:21,741 INFO org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater: master,60000,1385363388874.timerUpdater exiting
191757 2013-11-25 18:06:21,755 ERROR org.apache.hadoop.hbase.master.HMaster: Region server ^@^@slave10,60020,1385363390188 reported a fatal error:
191758 ABORTING region server slave10,60020,1385363390188: Unrecoverable exception while closing region productdevice,20131122-1-354890041701600,1385348706791.a587f1a15b4a3b10fc0e87       a804487532., still finishing close
191759 Cause:
191760 org.apache.hadoop.hbase.DroppedSnapshotException: region: productdevice,20131122-1-354890041701600,1385348706791.a587f1a15b4a3b10fc0e87a804487532.
191761     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
191762     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
191763     at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
191764     at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
191765     at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
191766     at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
191767     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
191768     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
191769     at java.lang.Thread.run(Thread.java:662)
191770 Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "slave10/192.168.1.210"; destination h       ost is: "master":8020; 
191771     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
191772     at org.apache.hadoop.ipc.Client.call(Client.java:1241)
191773     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
191774     at $Proxy16.getFileInfo(Unknown Source)
191775     at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
191776     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
191777     at java.lang.reflect.Method.invoke(Method.java:597)
191778     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
191779     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
191780     at $Proxy16.getFileInfo(Unknown Source)
191781     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
191782     at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
191783     at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
191784     at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
191785     at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
191786     at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:852)
191787     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:924)
191788     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:904)
191789     at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:805)
191790     at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746)
191791     at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2348)
191792     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1580)
191793     ... 8 more
191794 Caused by: java.io.IOException: Connection reset by peer
191795     at sun.nio.ch.FileDispatcher.read0(Native Method)
191796     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
191797     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
191798     at sun.nio.ch.IOUtil.read(IOUtil.java:171)
191799     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
191800     at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:56)
191801     at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
191803     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
191804     at java.io.FilterInputStream.read(FilterInputStream.java:116)
191805     at java.io.FilterInputStream.read(FilterInputStream.java:116)
191806     at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:420)
191807     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
191808     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
191809     at java.io.FilterInputStream.read(FilterInputStream.java:66)
191810     at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:276)
191811     at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
191812     at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
191813     at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
191814     at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:985)
191815     at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:948)
191816     at org.apache.hadoop.ipc.Client$Connection.run(Client.java:846)
191817 
191818 2013-11-25 18:06:21,762 ERROR org.apache.hadoop.hbase.master.HMaster: Region server ^@^@slave02,60020,1385363390113 reported a fatal error:
191819 ABORTING region server slave02,60020,1385363390113: Unrecoverable exception while closing region bitmap_resolution,,1385060184009.75ff45ea678aa0698d79c31a00a76d65., still fin       ishing close
191820 Cause:
191821 org.apache.hadoop.hbase.DroppedSnapshotException: region: bitmap_resolution,,1385060184009.75ff45ea678aa0698d79c31a00a76d65.
191822     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
191823     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
191824     at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
191825     at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
191826     at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
191827     at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
191828     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
191829     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
191830     at java.lang.Thread.run(Thread.java:662)
191831 Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "slave02/192.168.1.202"; destination h       ost is: "master":8020; 
191832     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
191833     at org.apache.hadoop.ipc.Client.call(Client.java:1241)
191834     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
191835     at $Proxy16.getFileInfo(Unknown Source)
191836     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
191837     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
191838     at java.lang.reflect.Method.invoke(Method.java:597)
191839     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
191840     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
191841     at $Proxy16.getFileInfo(Unknown Source)
191842     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
191843     at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
191844     at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
191845     at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
191846     at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
191847     at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:852)
191848     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:924)
191849     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:904)
191850     at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:805)
191851     at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746)
191852     at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2348)
191853     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1580)
191854     ... 8 more
191855 Caused by: java.io.IOException: Connection reset by peer
191856     at sun.nio.ch.FileDispatcher.read0(Native Method)
191857     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
191858     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
191859     at sun.nio.ch.IOUtil.read(IOUtil.java:171)
191860     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
191861     at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:56)
191862     at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
191863     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
191864     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
191865     at java.io.FilterInputStream.read(FilterInputStream.java:116)
191866     at java.io.FilterInputStream.read(FilterInputStream.java:116)
191867     at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:420)
191868     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
191869     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
191870     at java.io.FilterInputStream.read(FilterInputStream.java:66)
191871     at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:276)
191872     at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
191873     at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
191874     at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
191875     at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:985)
191876     at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:948)
191877     at org.apache.hadoop.ipc.Client$Connection.run(Client.java:846)
191878 
191879 2013-11-25 18:06:21,770 ERROR org.apache.hadoop.hbase.master.HMaster: Region server ^@^@slave02,60020,1385363390113 reported a fatal error:
191880 ABORTING region server slave02,60020,1385363390113: Unrecoverable exception while closing region productdevicehour,2013112119-1-354890041728435,1385354939937.849a12ee43543851       23611342e154bd36., still finishing close
191881 Cause:
191882 org.apache.hadoop.hbase.DroppedSnapshotException: region: productdevicehour,2013112119-1-354890041728435,1385354939937.849a12ee4354385123611342e154bd36.
191883     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
191884     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
191885     at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
191886     at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
191887     at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
191888     at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
191889     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
191890     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
191891     at java.lang.Thread.run(Thread.java:662)
191892 Caused by: java.net.ConnectException: Call From slave02/192.168.1.202 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more d       etails see:  http://wiki.apache.org/hadoop/ConnectionRefused
191893     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)                                                                                         
191894     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
191895     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
191896     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
191897     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
191898     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
191899     at org.apache.hadoop.ipc.Client.call(Client.java:1241)
191900     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
191901     at $Proxy16.getFileInfo(Unknown Source)
191902     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
191903     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
191904     at java.lang.reflect.Method.invoke(Method.java:597)
191905     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
191906     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
191907     at $Proxy16.getFileInfo(Unknown Source)
191908     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
191909     at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
191910     at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
191911     at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
191912     at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
191913     at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:852)
191914     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:924)
191915     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:904)
191916     at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:805)
191917     at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746)
191918     at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2348)
191919     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1580)
191920     ... 8 more
191921 Caused by: java.net.ConnectException: Connection refused
191922     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
191923     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
191924     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
191925     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
191926     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
191927     at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:509)
191928     at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:603)
191929     at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
191930     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1290)
191931     at org.apache.hadoop.ipc.Client.call(Client.java:1208)
191932     ... 28 more
191933 
191934 2013-11-25 18:06:21,783 ERROR org.apache.hadoop.hbase.master.HMaster: Region server ^@^@slave09,60020,1385363389867 reported a fatal error:
191935 ABORTING region server slave09,60020,1385363389867: Unrecoverable exception while closing region duration,20131122-1-1-2.3-bde52fc513ee4675b996a0e0d934c581,1385108566990.80b8       0663d05c92a57a38c28038fa67a2., still finishing close
191936 Cause:
191937 org.apache.hadoop.hbase.DroppedSnapshotException: region: duration,20131122-1-1-2.3-bde52fc513ee4675b996a0e0d934c581,1385108566990.80b80663d05c92a57a38c28038fa67a2.
191938     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
191939     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
191940     at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
191941     at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
191942     at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
191943     at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
191944     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
191945     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
191946     at java.lang.Thread.run(Thread.java:662)
191947 Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "slave09/192.168.1.209"; destination h       ost is: "master":8020; 
191948     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
191949     at org.apache.hadoop.ipc.Client.call(Client.java:1241)
191950     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
191951     at $Proxy16.getFileInfo(Unknown Source)
191952     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
191953     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
191954     at java.lang.reflect.Method.invoke(Method.java:597)
191955     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
191956     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
191957     at $Proxy16.getFileInfo(Unknown Source)
191958     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
191959     at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
191960     at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
191961     at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
191962     at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
191963     at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:852)
191964     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:924)
191965     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:904)
191966     at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:805)
191967     at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746)
191968     at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2348)
191969     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1580)
191970     ... 8 more
191971 Caused by: java.io.IOException: Connection reset by peer
191972     at sun.nio.ch.FileDispatcher.read0(Native Method)
191973     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
191974     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
191975     at sun.nio.ch.IOUtil.read(IOUtil.java:171)
191976     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
191977     at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:56)
191978     at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
191979     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
191980     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
191981     at java.io.FilterInputStream.read(FilterInputStream.java:116)
191982     at java.io.FilterInputStream.read(FilterInputStream.java:116)
191983     at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:420)
191984     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
191985     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
191986     at java.io.FilterInputStream.read(FilterInputStream.java:66)
 2013-11-25 18:06:21,783 ERROR org.apache.hadoop.hbase.master.HMaster: Region server ^@^@slave09,60020,1385363389867 reported a fatal error:
191996 ABORTING region server slave09,60020,1385363389867: Unrecoverable exception while closing region event,1385351227552-25231,1385355740449.65d06b2edd8d01fe9c3e2c2b5c8f745a., st       ill finishing close
191997 Cause:
191998 org.apache.hadoop.hbase.DroppedSnapshotException: region: event,1385351227552-25231,1385355740449.65d06b2edd8d01fe9c3e2c2b5c8f745a.
191999     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
192000     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
192001     at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
192002     at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
192003     at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
192004     at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
192005     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
192006     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
192007     at java.lang.Thread.run(Thread.java:662)
192008 Caused by: java.net.ConnectException: Call From slave09/192.168.1.209 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more d       etails see:  http://wiki.apache.org/hadoop/ConnectionRefused
192009     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
192010     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
192011     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
192012     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
192013     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
192014     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
192015     at org.apache.hadoop.ipc.Client.call(Client.java:1241)
192016     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
192017     at $Proxy16.getFileInfo(Unknown Source)
192018     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
192019     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
192020     at java.lang.reflect.Method.invoke(Method.java:597)
192021     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
192022     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
192023     at $Proxy16.getFileInfo(Unknown Source)
192024     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
192025     at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
192026     at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
192027     at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
192028     at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
192029     at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:852)
192030     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:924)
192031     at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:904)
192032     at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:805)
192033     at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746)
192034     at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2348)
192035     at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1580)
192036     ... 8 more
192037 Caused by: java.net.ConnectException: Connection refused
192038     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
192039     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
192040     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)


Logs from Slave

2013-11-25 18:06:21,982 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed TestTable,0009454723,1385103107169.f5cf4e66dce7cb12666e0c835de37f3e.
2013-11-25 18:06:21,987 INFO org.apache.hadoop.hbase.regionserver.Store: Closed f
2013-11-25 18:06:21,988 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed event,1385068619365-17160,1385127551266.3e19a908335088a2884b9d6ce6e63e64.
2013-11-25 18:06:21,988 INFO org.apache.hadoop.hbase.regionserver.HRegion: Running close preflush of bitmap_os,,1385046580329.84afdd67c81afc8bcd7ddeff1535edaa.
2013-11-25 18:06:21,989 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server slave03,60020,1385363389944: Unrecoverable exception while closing region bitmap_carrier,1#Vodafone#354890041727485,1385118856342.3e38b34fd58bdb9a500b506aaf1f3253., still finishing close
org.apache.hadoop.hbase.DroppedSnapshotException: region: bitmap_carrier,1#Vodafone#354890041727485,1385118856342.3e38b34fd58bdb9a500b506aaf1f3253.
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Call From slave03/192.168.1.203 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
        at org.apache.hadoop.ipc.Client.call(Client.java:1241)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at $Proxy16.getFileInfo(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at $Proxy16.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
        at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:852)
        at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:924)
        at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:904)
        at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:805)
        at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746)
        at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2348)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1580)
        ... 8 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:509)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:603)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1290)
        at org.apache.hadoop.ipc.Client.call(Client.java:1208)
        ... 28 more
2013-11-25 18:06:21,990 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [coprocessor.EndPoint_SA]
2013-11-25 18:06:21,993 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=31, numberOfStores=31, numberOfStorefiles=36, storefileIndexSizeMB=1, rootIndexSizeKB=1207, totalStaticIndexSizeKB=7530, totalStaticBloomSizeKB=0, memstoreSizeMB=223, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=1998084, writeRequestsCount=456, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=484, maxHeapMB=1019, blockCacheSizeMB=144.62, blockCacheFreeMB=110.35, blockCacheCount=2261, blockCacheHitCount=2030544, blockCacheMissCount=38759, blockCacheEvictedCount=18051, blockCacheHitRatio=98%, blockCacheHitCachingRatio=98%, hdfsBlocksLocalityIndex=49, slowHLogAppendCount=0, fsReadLatencyHistogramMean=2562237.51, fsReadLatencyHistogramCount=4740.00, fsReadLatencyHistogramMedian=267495.00, fsReadLatencyHistogram75th=311775.00, fsReadLatencyHistogram95th=581948.70, fsReadLatencyHistogram99th=3633498.98, fsReadLatencyHistogram999th=13829725.88, fsPreadLatencyHistogramMean=4629422.78, fsPreadLatencyHistogramCount=18090.00, fsPreadLatencyHistogramMedian=1794208.00, fsPreadLatencyHistogram75th=3066611.00, fsPreadLatencyHistogram95th=15731207.10, fsPreadLatencyHistogram99th=28249995.59, fsPreadLatencyHistogram999th=92628744.50, fsWriteLatencyHistogramMean=332585.56, fsWriteLatencyHistogramCount=3111.00, fsWriteLatencyHistogramMedian=223809.00, fsWriteLatencyHistogram75th=283576.00, fsWriteLatencyHistogram95th=516580.80, fsWriteLatencyHistogram99th=4027526.64, fsWriteLatencyHistogram999th=12034060.39
2013-11-25 18:06:22,001 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unrecoverable exception while closing region bitmap_carrier,1#Vodafone#354890041727485,1385118856342.3e38b34fd58bdb9a500b506aaf1f3253., still finishing close
2013-11-25 18:06:22,002 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_CLOSE_REGION
java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: bitmap_carrier,1#Vodafone#354890041727485,1385118856342.3e38b34fd58bdb9a500b506aaf1f3253.
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:133)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: bitmap_carrier,1#Vodafone#354890041727485,1385118856342.3e38b34fd58bdb9a500b506aaf1f3253.
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        ... 4 more
Caused by: java.net.ConnectException: Call From slave03/192.168.1.203 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
        at org.apache.hadoop.ipc.Client.call(Client.java:1241)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at $Proxy16.getFileInfo(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at $Proxy16.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
        at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:852)
        at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:924)
        at org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:904)
        at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:805)
        at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746)
        at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2348)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1580)
        ... 8 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:509)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:603)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1290)
        at org.apache.hadoop.ipc.Client.call(Client.java:1208)
        ... 28 more
2013-11-25 18:06:22,002 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2013-11-25 18:06:22,003 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting
2013-11-25 18:06:22,003 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020: exiting
2013-11-25 18:06:22,003 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting

2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 0 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 2 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020: exiting
2013-11-25 18:06:22,003 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting
2013-11-25 18:06:22,004 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2013-11-25 18:06:22,006 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2013-11-25 18:06:22,008 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
2013-11-25 18:06:22,008 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker slave03,60020,1385363389944 exiting
2013-11-25 18:06:22,019 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server slave03,60020,1385363389944: Unrecoverable exception while closing region error,1385001385292-18030,1385053818394.d1566031ca7c7df9fbd7e2f8e29f1d56., still finishing close
java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-25 18:06:22,019 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [coprocessor.EndPoint_SA]
2013-11-25 18:06:22,021 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=31, numberOfStores=31, numberOfStorefiles=36, storefileIndexSizeMB=1, rootIndexSizeKB=1207, totalStaticIndexSizeKB=7530, totalStaticBloomSizeKB=0, memstoreSizeMB=223, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=1998084, writeRequestsCount=456, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=485, maxHeapMB=1019, blockCacheSizeMB=144.62, blockCacheFreeMB=110.35, blockCacheCount=2261, blockCacheHitCount=2030544, blockCacheMissCount=38759, blockCacheEvictedCount=18051, blockCacheHitRatio=98%, blockCacheHitCachingRatio=98%, hdfsBlocksLocalityIndex=49, slowHLogAppendCount=0, fsReadLatencyHistogramMean=2562237.51, fsReadLatencyHistogramCount=4740.00, fsReadLatencyHistogramMedian=267495.00, fsReadLatencyHistogram75th=311775.00, fsReadLatencyHistogram95th=581948.70, fsReadLatencyHistogram99th=3633498.98, fsReadLatencyHistogram999th=13829725.88, fsPreadLatencyHistogramMean=4629422.78, fsPreadLatencyHistogramCount=18090.00, fsPreadLatencyHistogramMedian=1794208.00, fsPreadLatencyHistogram75th=3066611.00, fsPreadLatencyHistogram95th=15731207.10, fsPreadLatencyHistogram99th=28249995.59, fsPreadLatencyHistogram999th=92628744.50, fsWriteLatencyHistogramMean=332585.56, fsWriteLatencyHistogramCount=3111.00, fsWriteLatencyHistogramMedian=223809.00, fsWriteLatencyHistogram75th=283576.00, fsWriteLatencyHistogram95th=516580.80, fsWriteLatencyHistogram99th=4027526.64, fsWriteLatencyHistogram999th=12034060.39
2013-11-25 18:06:22,023 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unrecoverable exception while closing region error,1385001385292-18030,1385053818394.d1566031ca7c7df9fbd7e2f8e29f1d56., still finishing close
2013-11-25 18:06:22,023 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_CLOSE_REGION
java.lang.RuntimeException: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:133)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-25 18:06:22,032 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [coprocessor.EndPoint_SA]
2013-11-25 18:06:22,034 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=31, numberOfStores=31, numberOfStorefiles=36, storefileIndexSizeMB=1, rootIndexSizeKB=1207, totalStaticIndexSizeKB=7530, totalStaticBloomSizeKB=0, memstoreSizeMB=223, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=1998084, writeRequestsCount=456, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=486, maxHeapMB=1019, blockCacheSizeMB=144.62, blockCacheFreeMB=110.35, blockCacheCount=2261, blockCacheHitCount=2030544, blockCacheMissCount=38759, blockCacheEvictedCount=18051, blockCacheHitRatio=98%, blockCacheHitCachingRatio=98%, hdfsBlocksLocalityIndex=49, slowHLogAppendCount=0, fsReadLatencyHistogramMean=2562237.51, fsReadLatencyHistogramCount=4740.00, fsReadLatencyHistogramMedian=267495.00, fsReadLatencyHistogram75th=311775.00, fsReadLatencyHistogram95th=581948.70, fsReadLatencyHistogram99th=3633498.98, fsReadLatencyHistogram999th=13829725.88, fsPreadLatencyHistogramMean=4629422.78, fsPreadLatencyHistogramCount=18090.00, fsPreadLatencyHistogramMedian=1794208.00, fsPreadLatencyHistogram75th=3066611.00, fsPreadLatencyHistogram95th=15731207.10, fsPreadLatencyHistogram99th=28249995.59, fsPreadLatencyHistogram999th=92628744.50, fsWriteLatencyHistogramMean=332585.56, fsWriteLatencyHistogramCount=3111.00, fsWriteLatencyHistogramMedian=223809.00, fsWriteLatencyHistogram75th=283576.00, fsWriteLatencyHistogram95th=516580.80, fsWriteLatencyHistogram99th=4027526.64, fsWriteLatencyHistogram999th=12034060.39
2013-11-25 18:06:22,040 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unrecoverable exception while closing region productdevice,20131119-1-354890041703320,1384887770169.71d9255f8376a930d04e0d15f982ccbe., still finishing close
2013-11-25 18:06:22,040 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_CLOSE_REGION
java.lang.RuntimeException: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:133)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        ... 4 more
2013-11-25 18:06:22,041 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server slave03,60020,1385363389944: Unrecoverable exception while closing region usinglog,1384920170789-23920,1384937680531.005c894966947d5d2bc48701ea620a49., still finishing close
java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-25 18:06:22,048 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [coprocessor.EndPoint_SA]
2013-11-25 18:06:22,049 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=31, numberOfStores=31, numberOfStorefiles=36, storefileIndexSizeMB=1, rootIndexSizeKB=1207, totalStaticIndexSizeKB=7530, totalStaticBloomSizeKB=0, memstoreSizeMB=223, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=1998084, writeRequestsCount=456, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=470, maxHeapMB=1019, blockCacheSizeMB=144.62, blockCacheFreeMB=110.35, blockCacheCount=2261, blockCacheHitCount=2030544, blockCacheMissCount=38759, blockCacheEvictedCount=18051, blockCacheHitRatio=98%, blockCacheHitCachingRatio=98%, hdfsBlocksLocalityIndex=49, slowHLogAppendCount=0, fsReadLatencyHistogramMean=2562237.51, fsReadLatencyHistogramCount=4740.00, fsReadLatencyHistogramMedian=267495.00, fsReadLatencyHistogram75th=311775.00, fsReadLatencyHistogram95th=581948.70, fsReadLatencyHistogram99th=3633498.98, fsReadLatencyHistogram999th=13829725.88, fsPreadLatencyHistogramMean=4629422.78, fsPreadLatencyHistogramCount=18090.00, fsPreadLatencyHistogramMedian=1794208.00, fsPreadLatencyHistogram75th=3066611.00, fsPreadLatencyHistogram95th=15731207.10, fsPreadLatencyHistogram99th=28249995.59, fsPreadLatencyHistogram999th=92628744.50, fsWriteLatencyHistogramMean=332585.56, fsWriteLatencyHistogramCount=3111.00, fsWriteLatencyHistogramMedian=223809.00, fsWriteLatencyHistogram75th=283576.00, fsWriteLatencyHistogram95th=516580.80, fsWriteLatencyHistogram99th=4027526.64, fsWriteLatencyHistogram999th=12034060.39
2013-11-25 18:06:22,051 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unrecoverable exception while closing region usinglog,1384920170789-23920,1384937680531.005c894966947d5d2bc48701ea620a49., still finishing close
2013-11-25 18:06:22,051 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_CLOSE_REGION
java.lang.RuntimeException: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:133)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        ... 4 more
2013-11-25 18:06:22,052 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server slave03,60020,1385363389944: Unrecoverable exception while closing region clientdata,1385137035557-42789,1385355563656.42ceb3de9c3a30d065f2c26fc4b8c2df., still finishing close
java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-25 18:06:22,053 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [coprocessor.EndPoint_SA]
2013-11-25 18:06:22,055 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=31, numberOfStores=31, numberOfStorefiles=36, storefileIndexSizeMB=1, rootIndexSizeKB=1207, totalStaticIndexSizeKB=7530, totalStaticBloomSizeKB=0, memstoreSizeMB=223, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=1998084, writeRequestsCount=456, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=470, maxHeapMB=1019, blockCacheSizeMB=144.62, blockCacheFreeMB=110.35, blockCacheCount=2261, blockCacheHitCount=2030544, blockCacheMissCount=38759, blockCacheEvictedCount=18051, blockCacheHitRatio=98%, blockCacheHitCachingRatio=98%, hdfsBlocksLocalityIndex=49, slowHLogAppendCount=0, fsReadLatencyHistogramMean=2562237.51, fsReadLatencyHistogramCount=4740.00, fsReadLatencyHistogramMedian=267495.00, fsReadLatencyHistogram75th=311775.00, fsReadLatencyHistogram95th=581948.70, fsReadLatencyHistogram99th=3633498.98, fsReadLatencyHistogram999th=13829725.88, fsPreadLatencyHistogramMean=4629422.78, fsPreadLatencyHistogramCount=18090.00, fsPreadLatencyHistogramMedian=1794208.00, fsPreadLatencyHistogram75th=3066611.00, fsPreadLatencyHistogram95th=15731207.10, fsPreadLatencyHistogram99th=28249995.59, fsPreadLatencyHistogram999th=92628744.50, fsWriteLatencyHistogramMean=332585.56, fsWriteLatencyHistogramCount=3111.00, fsWriteLatencyHistogramMedian=223809.00, fsWriteLatencyHistogram75th=283576.00, fsWriteLatencyHistogram95th=516580.80, fsWriteLatencyHistogram99th=4027526.64, fsWriteLatencyHistogram999th=12034060.39
2013-11-25 18:06:22,056 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unrecoverable exception while closing region clientdata,1385137035557-42789,1385355563656.42ceb3de9c3a30d065f2c26fc4b8c2df., still finishing close
2013-11-25 18:06:22,056 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_CLOSE_REGION
java.lang.RuntimeException: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:133)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        ... 4 more
2013-11-25 18:06:22,057 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server slave03,60020,1385363389944: Unrecoverable exception while closing region usinglog,1384893879578-27094,1384905488699.8f1d80c344a2359c9779d4bc0210dec7., still finishing close
java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-25 18:06:22,057 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [coprocessor.EndPoint_SA]
2013-11-25 18:06:22,058 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=31, numberOfStores=31, numberOfStorefiles=36, storefileIndexSizeMB=1, rootIndexSizeKB=1207, totalStaticIndexSizeKB=7530, totalStaticBloomSizeKB=0, memstoreSizeMB=223, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=1998084, writeRequestsCount=456, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=471, maxHeapMB=1019, blockCacheSizeMB=144.62, blockCacheFreeMB=110.35, blockCacheCount=2261, blockCacheHitCount=2030544, blockCacheMissCount=38759, blockCacheEvictedCount=18051, blockCacheHitRatio=98%, blockCacheHitCachingRatio=98%, hdfsBlocksLocalityIndex=49, slowHLogAppendCount=0, fsReadLatencyHistogramMean=2562237.51, fsReadLatencyHistogramCount=4740.00, fsReadLatencyHistogramMedian=267495.00, fsReadLatencyHistogram75th=311775.00, fsReadLatencyHistogram95th=581948.70, fsReadLatencyHistogram99th=3633498.98, fsReadLatencyHistogram999th=13829725.88, fsPreadLatencyHistogramMean=4629422.78, fsPreadLatencyHistogramCount=18090.00, fsPreadLatencyHistogramMedian=1794208.00, fsPreadLatencyHistogram75th=3066611.00, fsPreadLatencyHistogram95th=15731207.10, fsPreadLatencyHistogram99th=28249995.59, fsPreadLatencyHistogram999th=92628744.50, fsWriteLatencyHistogramMean=332585.56, fsWriteLatencyHistogramCount=3111.00, fsWriteLatencyHistogramMedian=223809.00, fsWriteLatencyHistogram75th=283576.00, fsWriteLatencyHistogram95th=516580.80, fsWriteLatencyHistogram99th=4027526.64, fsWriteLatencyHistogram999th=12034060.39
2013-11-25 18:06:22,059 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unrecoverable exception while closing region usinglog,1384893879578-27094,1384905488699.8f1d80c344a2359c9779d4bc0210dec7., still finishing close
2013-11-25 18:06:22,059 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_CLOSE_REGION
java.lang.RuntimeException: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:133)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Aborting flush because server is abortted...
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1496)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1008)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        ... 4 more
2013-11-25 18:06:22,060 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server slave03,60020,1385363389944: Unrecoverable exception while closing region bitmap_os,,1385046580329.84afdd67c81afc8bcd7ddeff1535edaa., still finishing close
org.apache.hadoop.hbase.DroppedSnapshotException: region: bitmap_os,,1385046580329.84afdd67c81afc8bcd7ddeff1535edaa.
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1605)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1479)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:992)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Call From slave03/192.168.1.203 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
        at org.apache.hadoop.ipc.Client.call(Client.java:1241)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at $Proxy16.getFileInfo(Unknown Source)


Thanks a lot.

Brad
Mime
View raw message