hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatesh <vramanatha...@aol.com>
Subject Region servers suddenly disappearing
Date Sun, 10 Oct 2010 21:13:15 GMT

 Some of the region servers suddenly dying..I've pasted relevant log lines..I don't see any
error in datanodes
Any ideas?
thanks
venkatsh

.....


 2010-10-10 12:55:36,664 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException:
Unable to create new block.        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2845)
       at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
       at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)

2010-10-10 12:55:36,664 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-8758558338582893960_95415
bad datanode[0] nodes == null
2010-10-10 12:55:36,665 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations.
Source file "/hbase_data/user_activity/compaction.dir/78194102/766401078063435042" - Aborting...
2010-10-10 12:55:36,666 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split
failed for region user_activity,1286729575294_11655_614aa74e,1286729678877
java.io.EOFException
        at java.io.DataInputStream.readByte(DataInputStream.java:250)        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
       at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
2010-10-10 12:55:40,176 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.EOFException
2010-10-10 12:55:40,176 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-568910271688144725_95415



2010-10-10 12:55:53,353 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed activities2010-10-10
12:55:53,353 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed user_activity,1286232613677_albridgew4_18363_c45677e1,12862335110072010-10-10
12:55:53,353 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: closing region user_activity,1286202422485_bayequip_15725_a6b7893e,12862031448812010-10-10
12:55:53,353 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing user_activity,1286202422485_bayequip_15725_a6b7893e,1286203144881:
disabling compactions & flushes2010-10-10 12:55:53,353 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on user_activity,1286202422485_bayequip_15725_a6b7893e,12862031448812010-10-10
12:55:53,353 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding
on region user_activity,1286202422485_15725_a6b7893e,12862031448812010-10-10 12:55:53,353
DEBUG org.apache.hadoop.hbase.regionserver.Store: closed activities2010-10-10 12:55:53,354
INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed user_activity,1286202422485_bayequip_15725_a6b7893e,1286203144881
2010-10-10 12:55:53,354 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: aborting
server at: 172.29.253.200:60020
2010-10-10 12:55:55,091 INFO org.apache.hadoop.hbase.Leases: regionserver/172.29.253.200:60020.leaseChecker
closing leases2010-10-10 12:55:55,091 INFO org.apache.hadoop.hbase.Leases: regionserver/172.29.253.200:60020.leaseChecker
closed leases
2010-10-10 12:55:59,664 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread
exiting
2010-10-10 12:55:59,664 INFO org.apache.zookeeper.ZooKeeper: Closing session: 0x22b967dce5d0001
2010-10-10 12:55:59,664 INFO org.apache.zookeeper.ClientCnxn: Closing ClientCnxn for session:
0x22b967dce5d0001
2010-10-10 12:55:59,669 INFO org.apache.zookeeper.ClientCnxn: Exception while closing send
thread for session 0x22b967dce5d0001 : Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
2010-10-10 12:55:59,775 INFO org.apache.zookeeper.ClientCnxn: Disconnecting ClientCnxn for
session: 0x22b967dce5d0001
2010-10-10 12:55:59,775 INFO org.apache.zookeeper.ZooKeeper: Session: 0x22b967dce5d0001 closed
2010-10-10 12:55:59,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection
with ZooKeeper
2010-10-10 12:55:59,775 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2010-10-10 12:55:59,776 ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file /hbase_data/user_activity/78194102/activities/8044918410206348854
: java.io.EOFException
java.io.EOFException        at java.io.DataInputStream.readByte(DataInputStream.java:250)
       at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message