hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-1087) DFS failures did not shutdown regionserver
Date Tue, 23 Dec 2008 18:36:46 GMT
DFS failures did not shutdown regionserver
------------------------------------------

                 Key: HBASE-1087
                 URL: https://issues.apache.org/jira/browse/HBASE-1087
             Project: Hadoop HBase
          Issue Type: Bug
    Affects Versions: 0.19.0
            Reporter: Jonathan Gray
            Priority: Critical
             Fix For: 0.19.0


I lost three Datanodes, reasons of which are still being investigated, but it has left a number
of regions unable to be written to.

Relevant logs:

2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException:
Connection reset by peer
        at sun.nio.ch.FileDispatcher.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
        at sun.nio.ch.IOUtil.write(IOUtil.java:93)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
        at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)

2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor
exception  for block blk_3615512604618056881_86411java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:197)
        at java.io.DataInputStream.readLong(DataInputStream.java:416)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2318)

2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_3615512604618056881_86411
bad datanode[0] 72.34.249.214:50010
2008-12-23 02:35:59,595 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_3615512604618056881_86411
in pipeline 72.34.249.214:50010, 72.34.249.213:50010, 72.34.249.219:50010: bad datanode 72.34.249.214:50010

2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-3678518999439029831_86910
2008-12-23 02:38:27,711 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
5392007859847346106 has been explicitly released by client
2008-12-23 02:38:30,048 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
-5905479324505886709 explicitly acquired by client
2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-226119866881174578_86911
2008-12-23 02:38:34,908 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
346704317670569896 explicitly acquired by client
2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_1719395740576248920_86913
2008-12-23 02:38:40,945 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
3819942931078736534 explicitly acquired by client
2008-12-23 02:38:45,572 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
254119037927296402 explicitly acquired by client
2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_2443399503093377808_86915
2008-12-23 02:38:49,092 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
8573046623144113301 explicitly acquired by client
2008-12-23 02:38:49,385 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
7686739650257547105 explicitly acquired by client
2008-12-23 02:38:49,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
5582966798894532276 explicitly acquired by client
2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException:
Unable to create new block.
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2723)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_2443399503093377808_86915
bad datanode[0] nodes == null
2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations.
Aborting...
2008-12-23 02:38:51,704 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append.
Requesting close of log
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
2008-12-23 02:38:51,706 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling
failed with ioe:
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
2008-12-23 02:38:51,706 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.net.ConnectException:
Connection refused


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message