hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Data node problem after reinstall
Date Tue, 01 Nov 2011 16:35:35 GMT
I don't see anything wrong in that log, do you actually have WARN or
ERROR level log lines? Those might be a better start.

Also please explain how you figured that the DN can only accept
replicas and nothing from HBase, hopefully with evidence. This will
greatly help.

Thx,

J-D

On Tue, Nov 1, 2011 at 12:28 AM, Weihua JIANG <weihua.jiang@gmail.com> wrote:
> Hi all,
>
> I am not sure whether it is a hbase problem or a hdfs problem.
>
> When have 8 datanodes & regionservers each with 12 disks. One data node is
> down due to its system disk broken. After replacing the disk and reinstall
> the OS, we tried to online this DN & region server. The region server is
> OK. But, the data node seems can only accept block replication from other
> data nodes, but failed to accept writing from HBase. Even after we removed
> all content of data dir and restarted the DN, this problem still exists.
> The log is:
> 2011-11-01 22:39:44,397 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6413273237261014407_77967 src: /10.1.2.23:56859 dest:
> /10.1.2.25:50010of size 67108864
> 2011-11-01 22:39:45,959 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6561707380479467846_78302 src: /10.1.2.20:47578 dest:
> /10.1.2.25:50010of size 67108864
> 2011-11-01 22:39:46,314 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest: /10.1.2.25:50010
> 2011-11-01 22:39:46,829 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.21:35450, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_1189047403, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_7603159785320952833_235458, duration: 20790059066
> 2011-11-01 22:39:46,829 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_7603159785320952833_235458 terminating
> 2011-11-01 22:39:46,850 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_7980572561428708688_235458 src: /10.1.2.19:37630 dest: /10.1.2.25:50010
> 2011-11-01 22:39:47,093 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.19:37611, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_-81251145, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_-9067419333696936066_235458, duration: 20478973800
> 2011-11-01 22:39:47,093 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_-9067419333696936066_235458 terminating
> 2011-11-01 22:39:47,117 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-495608789656244385_235458 src: /10.1.2.21:35468 dest: /10.1.2.25:50010
> 2011-11-01 22:39:47,185 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.23:56845, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_1647768037, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_-7935479347615843321_235458, duration: 20595122675
> 2011-11-01 22:39:47,185 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_-7935479347615843321_235458 terminating
> 2011-11-01 22:39:47,325 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest:
> /10.1.2.25:50010of size 67108864
> 2011-11-01 22:39:49,981 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.25:60888, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_-1345645880, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_2614344148027463064_235458, duration: 19952693395
> 2011-11-01 22:39:49,982 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for
> block blk_2614344148027463064_235458 terminating
> 2011-11-01 22:39:49,999 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_5157459941530714338_235458 src: /10.1.2.25:60892 dest: /10.1.2.25:50010
> 2011-11-01 22:39:51,258 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest: /10.1.2.25:50010
> 2011-11-01 22:39:51,259 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest:
> /10.1.2.25:50010of size 124
> 2011-11-01 22:39:52,357 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-5980257649446708968_78270 src: /10.1.2.23:56867 dest: /10.1.2.25:50010
> 2011-11-01 22:39:53,336 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-7734816649100892246_235458 src: /10.1.2.26:36320 dest: /10.1.2.25:50010
> 2011-11-01 22:39:57,162 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.22:37400, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_-858857344, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_-1542408817127580486_235458, duration: 21156605250
> 2011-11-01 22:39:57,162 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_-1542408817127580486_235458 terminating
>
> Thanks
> Weihua
>

Mime
View raw message