hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuti Awasthi <stutiawas...@hcl.com>
Subject RE: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append
Date Fri, 14 Oct 2011 09:17:01 GMT
Hi Lars,
I am using Hbase 0.90.3 and there was no inserting done on the nodes at that time.

-----Original Message-----
From: lars hofhansl [mailto:lhofhansl@yahoo.com] 
Sent: Friday, October 14, 2011 10:46 AM
To: user@hbase.apache.org
Subject: Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append

Did you do some heavy inserting during that time? I assume you use 0.90.x(?)

Might be HBASE-4387.

-- Lars



----- Original Message -----
From: Stuti Awasthi <stutiawasthi@hcl.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Cc: 
Sent: Thursday, October 13, 2011 9:44 PM
Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append

Hi Friends
I am using a mini distributed cluster of 3 nodes. Regionservers are running on all of 3 nodes.
This cluster is operational from 2 weeks now without any issue. Today morning I saw one of
the region servers are down.
I restarted the regionserver dynamically and it started fine and master also distributed regions
to this region server.
I looked at the log but not much understood what was the problem. Logs state that one of my
datanodes are bad, but when I checked, all datanodes are working fine.

Please suggest

Regionserver Log :

2011-10-13 14:35:47,974 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
java.io.IOException: Reflection
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1477)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2549)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
        ... 10 more
Caused by: java.io.IOException: DFSOutputStream is closed
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
        at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
        ... 14 more
2011-10-13 14:35:47,999 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter:
Using syncFs -- HDFS-200
2011-10-13 14:35:48,001 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING
region server serverName=master,60020,1316669478524, load=(requests=0, regions=4, usedHeap=31,
maxHeap=993): IOE in log roller
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0, memstoreSize=0, compactionQueueSize=0,
flushQueueSize=0, usedHeap=31, maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440,
blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9, blockCacheEvictedCount=0,
blockCacheHitRatio=99, blockCacheHitCachingRatio=99
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
IOE in log roller
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.



::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named
recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or
opinions presented in this email are solely those of the author and may not necessarily reflect
the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and
/ or publication of this message without the prior written consent of the author of this e-mail
is strictly prohibited. If you have received this email in error please delete it and notify
the sender immediately. Before opening any mail and attachments please check them for viruses
and defect.

-----------------------------------------------------------------------------------------------------------------------


Mime
View raw message