hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From llpind <sonny_h...@hotmail.com>
Subject Re: HBase looses regions.
Date Tue, 26 May 2009 17:06:15 GMT

All,

I tailed one of the region servers that seems to be handling most of the
load.   Here is what I saw:

2009-05-26 10:00:21,015 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:21,419 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:21,823 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:22,227 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:22,631 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:23,035 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:23,439 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:23,708 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.net.SocketTimeoutException: 5000 millis timeout while
waiting for channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.240.175:53828
remote=/192.168.240.179:50010]
        at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162)
        at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
        at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)

2009-05-26 10:00:23,708 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1087586329384370907_1077 bad datanode[0]
192.168.240.179:50010
2009-05-26 10:00:23,843 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:24,247 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:24,651 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:25,055 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:25,459 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:25,863 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:26,267 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:26,671 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:27,075 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:27,479 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:27,883 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:28,287 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:28,692 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:29,096 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:29,500 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:29,904 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:30,308 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:30,712 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:31,116 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:31,520 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:31,924 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:32,328 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:32,732 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:33,136 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:33,540 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:33,944 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:34,332 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:34,348 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:34,736 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:34,752 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:35,140 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:35,156 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:35,544 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:35,560 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:35,948 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:35,964 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:36,352 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:36,368 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:36,756 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:36,772 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:37,160 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:37,176 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:37,564 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:37,580 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:37,968 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:37,984 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:38,372 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:38,388 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:38,776 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:38,792 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:39,180 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:39,196 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:39,584 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:39,600 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:39,988 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:40,004 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:40,392 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:40,408 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:40,796 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:40,812 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:41,200 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:41,216 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:41,604 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:41,620 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:42,008 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:42,024 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:42,412 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:42,464 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:42,816 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:42,868 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:43,220 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:43,272 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:43,624 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:43,676 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:44,028 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:44,080 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:44,432 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:44,485 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:44,837 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:44,888 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:45,241 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:45,293 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:45,645 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:45,697 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:46,049 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:46,101 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:46,453 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:46,505 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:46,857 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:46,909 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:47,261 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:47,501 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:47,665 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:47,905 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:48,069 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:48,309 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:48,473 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:48,713 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:48,877 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:49,117 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:49,281 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:49,521 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:49,685 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:49,925 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:50,089 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:50,329 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:50,493 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:50,733 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:50,897 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:51,137 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:51,301 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:51,541 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:51,705 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:51,945 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:52,109 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:52,349 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:52,513 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:52,753 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:52,917 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:53,157 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:53,321 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:53,561 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:53,725 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:53,965 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:54,133 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:54,369 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:54,537 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:54,773 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:54,941 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:55,177 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:55,345 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:55,581 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:55,749 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:55,985 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:56,153 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:56,389 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:56,557 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:56,793 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:56,961 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:57,317 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:57,365 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:57,721 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:57,769 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:58,125 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:58,173 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:58,529 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:58,577 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:58,933 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:58,981 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:59,337 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:59,385 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:00:59,741 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:00:59,789 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:00,145 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:00,193 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:00,549 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:00,597 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:00,954 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:01,002 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:01,358 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:01,406 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:01,762 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:01,810 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:02,166 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:02,214 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:02,570 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:02,618 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:02,974 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:03,022 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:03,378 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:03,426 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:03,782 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473
retrying...
2009-05-26 10:01:03,830 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index
retrying...
2009-05-26 10:01:03,849 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split
failed for region tableA,ROW_KEY,1243357190459
java.io.IOException: All datanodes 192.168.240.179:50010 are bad.
Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2444)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
2009-05-26 10:01:03,891 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://ats181:54310/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357129598,
entries=100002. New log writer:
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357263846
2009-05-26 10:01:03,891 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357078715 whose
highest sequence/edit id is 1175008
2009-05-26 10:01:37,493 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://ats181:54310/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473,
entries=105000. New log writer:
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357297485
2009-05-26 10:01:39,353 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting  compaction on region tableA,ROW_KEY,1243357190459
2009-05-26 10:01:40,903 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://ats181:54310/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357263846,
entries=100001. New log writer:
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357300890
2009-05-26 10:01:40,904 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357129598 whose
highest sequence/edit id is 1275010
2009-05-26 10:01:40,912 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 whose
highest sequence/edit id is 1380010
2009-05-26 10:01:59,165 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...
2009-05-26 10:01:59,569 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...
2009-05-26 10:01:59,973 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...
2009-05-26 10:02:00,377 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...
2009-05-26 10:02:00,781 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...
2009-05-26 10:02:01,185 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...
2009-05-26 10:02:01,589 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...
2009-05-26 10:02:01,993 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file
/hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data
retrying...

-- 
View this message in context: http://www.nabble.com/HBase-looses-regions.-tp23657983p23727376.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message