hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-930) RegionServer stuck: HLog: Could not append. Requesting close of log java.io.IOException: Could not get block locations. Aborting...
Date Wed, 15 Oct 2008 20:58:46 GMT
RegionServer stuck: HLog: Could not append. Requesting close of log java.io.IOException: Could
not get block locations. Aborting...
-----------------------------------------------------------------------------------------------------------------------------------

                 Key: HBASE-930
                 URL: https://issues.apache.org/jira/browse/HBASE-930
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: stack


HDFS went wonky.  Manifest itself in regionserver with below: i.e. first can't replicate and
then the exceptions every time we append and then every time we try to rotate the log file.
  Exception is always: "IOException: Could not get block locations. Aborting..."  Meantime
the HDFS is fixed but for whatever reason the HRS just keeps on with the below failings; as
though the error is stuck in the DFSClient.

{code}
...
2008-10-15 02:54:59,867 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Completed compaction
of 1399814750/alternate_url store size is 2.9m
2008-10-15 02:54:59,868 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region enwiki_old,7qE8TUI5v-NIZu_9oZhaJF==,1221856888271 in 38sec
2008-10-15 02:59:08,507 INFO org.apache.hadoop.dfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /hbase/aa0-000-8.u.powerset.com/log_208.76.45.180_1223616861290_60020/hlog.dat.1224038569764
could only be re
plicated to 0 nodes, instead of 1
        at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1117)
        at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

        at org.apache.hadoop.ipc.Client.call(Client.java:715)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
        at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2440)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2323)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)

...
2008-10-15 02:59:11,357 WARN org.apache.hadoop.dfs.DFSClient: NotReplicatedYetException sleeping
/hbase/aa0-000-8.u.powerset.com/log_208.76.45.180_1223616861290_60020/hlog.dat.1224038569764
retries left 1
2008-10-15 02:59:14,566 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /hbase/aa0-000-8.u.powerset.com/log_208.76.45.180_1223616861290_60020/hlog.dat.122403
8569764 could only be replicated to 0 nodes, instead of 1
        at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1117)
        at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

        at org.apache.hadoop.ipc.Client.call(Client.java:715)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
        at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2440)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2323)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)

2008-10-15 02:59:14,566 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery for block null
bad datanode[0]
2008-10-15 02:59:14,566 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append.
Requesting close of log
java.io.IOException: Could not get block locations. Aborting...
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
2008-10-15 02:59:14,567 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog.
Number of entries: 81
2008-10-15 02:59:14,567 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020,
call batchUpdate([B@3e4afee5, row => Dnsw4RukHJ5-0FQddY35mF==, {column => misc:upload_time,
value => '...', column => page:mime, value => '...', c
olumn => page:content, value => '...', column => page:url, value => '...'}, -1)
from 208.76.44.183:59031: error: java.io.IOException: Could not get block locations. Aborting...
java.io.IOException: Could not get block locations. Aborting...
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
2008-10-15 02:59:14,588 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling
failed
java.io.IOException: Could not get block locations. Aborting...
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
...
{code}

For now adding abort of regionserver if we can't close the log roller file.  If we can't close,
then we're losing edits.  If this behavior causes us close to much, then need to dig in more.

Happened on pset cluster running 0.18.1 hadoop and 0.18.0 hbase.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message