hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1087) DFS failures did not shutdown regionserver
Date Wed, 24 Dec 2008 00:06:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659004#action_12659004
] 

stack commented on HBASE-1087:
------------------------------

I would have thought that '2008-12-23 02:38:51,704 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of log' should have triggered a hrs shutdown.

> DFS failures did not shutdown regionserver
> ------------------------------------------
>
>                 Key: HBASE-1087
>                 URL: https://issues.apache.org/jira/browse/HBASE-1087
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>            Reporter: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.19.0
>
>
> I lost three Datanodes, reasons of which are still being investigated, but it has left
a number of regions unable to be written to.
> Relevant logs:
> 2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception:
java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcher.write0(Native Method)
>         at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
>         at sun.nio.ch.IOUtil.write(IOUtil.java:93)
>         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
>         at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
>         at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>         at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>         at java.io.DataOutputStream.write(DataOutputStream.java:107)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)
> 2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor
exception  for block blk_3615512604618056881_86411java.io.EOFException
>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>         at java.io.DataInputStream.readLong(DataInputStream.java:416)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2318)
> 2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block
blk_3615512604618056881_86411 bad datanode[0] 72.34.249.214:50010
> 2008-12-23 02:35:59,595 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block
blk_3615512604618056881_86411 in pipeline 72.34.249.214:50010, 72.34.249.213:50010, 72.34.249.219:50010:
bad datanode 72.34.249.214:50010
> 2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
> 2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-3678518999439029831_86910
> 2008-12-23 02:38:27,711 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock 5392007859847346106 has been explicitly released by client
> 2008-12-23 02:38:30,048 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock -5905479324505886709 explicitly acquired by client
> 2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
> 2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-226119866881174578_86911
> 2008-12-23 02:38:34,908 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock 346704317670569896 explicitly acquired by client
> 2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
> 2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_1719395740576248920_86913
> 2008-12-23 02:38:40,945 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock 3819942931078736534 explicitly acquired by client
> 2008-12-23 02:38:45,572 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock 254119037927296402 explicitly acquired by client
> 2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
> 2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_2443399503093377808_86915
> 2008-12-23 02:38:49,092 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock 8573046623144113301 explicitly acquired by client
> 2008-12-23 02:38:49,385 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock 7686739650257547105 explicitly acquired by client
> 2008-12-23 02:38:49,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row
lock 5582966798894532276 explicitly acquired by client
> 2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception:
java.io.IOException: Unable to create new block.
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2723)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block
blk_2443399503093377808_86915 bad datanode[0] nodes == null
> 2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations.
Aborting...
> 2008-12-23 02:38:51,704 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append.
Requesting close of log
> java.net.ConnectException: Connection refused
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-23 02:38:51,706 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling
failed with ioe:
> java.net.ConnectException: Connection refused
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-23 02:38:51,706 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.net.ConnectException:
Connection refused

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message