Return-Path: X-Original-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4DF12974F for ; Thu, 5 Jan 2012 18:07:22 +0000 (UTC) Received: (qmail 6016 invoked by uid 500); 5 Jan 2012 18:07:19 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 5526 invoked by uid 500); 5 Jan 2012 18:07:18 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 5341 invoked by uid 99); 5 Jan 2012 18:07:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jan 2012 18:07:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of atm@cloudera.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jan 2012 18:07:13 +0000 Received: by vbbfa15 with SMTP id fa15so780449vbb.35 for ; Thu, 05 Jan 2012 10:06:52 -0800 (PST) Received: by 10.52.70.146 with SMTP id m18mr1366668vdu.127.1325786812307; Thu, 05 Jan 2012 10:06:52 -0800 (PST) MIME-Version: 1.0 Received: by 10.220.7.70 with HTTP; Thu, 5 Jan 2012 10:06:21 -0800 (PST) In-Reply-To: <1542FA4EE20C5048A5C2A3663BED2A6B0F88473D@szxeml531-mbx.china.huawei.com> References: <1542FA4EE20C5048A5C2A3663BED2A6B0F88473D@szxeml531-mbx.china.huawei.com> From: "Aaron T. Myers" Date: Thu, 5 Jan 2012 10:06:21 -0800 Message-ID: Subject: Re: Timeouts in Datanodes while block scanning To: hdfs-dev@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf307f3714fedb5304b5cbcb1e --20cf307f3714fedb5304b5cbcb1e Content-Type: text/plain; charset=ISO-8859-1 What version of HDFS? This question might be more appropriate for hdfs-user@ . -- Aaron T. Myers Software Engineer, Cloudera On Thu, Jan 5, 2012 at 8:59 AM, Uma Maheswara Rao G wrote: > Hi, > > I have 10 Node cluster running from last 25days( running with Hbase > cluster). Recently observed that for every continuos blocks scans, there > are many timeouts coming in DataNode. > After this block scan verifications, again reads succeeded. This > situation keep occurring many times now, for every continuous block scans. > Here Hbase continuously performing many random reads. > > Whether any one faced this situation in your clusters? > > Below is the logs with timeouts. > 2011-12-28 11:30:42,618 INFO DataNode.clienttrace > (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: / > 107.252.175.3:52764, bytes: 264192, op: HDFS_READ, cliID: > DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27, > srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid: > blk_1323251633953_187190 > 2011-12-28 11:30:42,621 INFO DataNode.clienttrace > (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: / > 107.252.175.3:52772, bytes: 396288, op: HDFS_READ, cliID: > DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27, > srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid: > blk_1323251635735_188342 > 2011-12-28 11:30:42,641 INFO DataNode.clienttrace > (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: / > 107.252.175.3:52796, bytes: 396288, op: HDFS_READ, cliID: > DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27, > srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid: > blk_1323251634096_187277 > 2011-12-28 11:30:42,889 INFO DataNode.clienttrace > (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: / > 107.252.175.3:52732, bytes: 264192, op: HDFS_READ, cliID: > DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27, > srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid: > blk_1323251635763_188363 > 2011-12-28 11:30:42,889 INFO DataNode.clienttrace > (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: / > 107.252.175.3:52637, bytes: 264192, op: HDFS_READ, cliID: > DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27, > srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid: > blk_1323251634921_187798 > 2011-12-28 11:30:42,976 INFO DataNode.clienttrace > (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: / > 107.252.175.3:52755, bytes: 396288, op: HDFS_READ, cliID: > DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27, > srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid: > blk_1323251635359_188075 > 2011-12-28 11:30:57,757 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251602823_167208 > 2011-12-28 11:32:15,757 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251599175_166755 > 2011-12-28 11:32:54,561 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251673745_194676 > 2011-12-28 11:33:33,561 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251640709_189383 > 2011-12-28 11:34:12,557 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251649630_190779 > 2011-12-28 11:34:51,557 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251463964_91885 > 2011-12-28 11:35:23,958 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251636310_188845 > 2011-12-28 11:36:01,155 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1322486683238_54999 > 2011-12-28 11:36:04,157 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251678959_195786 > 2011-12-28 11:36:43,157 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251641803_189561 > 2011-12-28 11:37:20,357 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1322486706170_66445 > 2011-12-28 11:37:44,759 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251646924_190359 > 2011-12-28 11:38:23,759 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251673776_194683 > 2011-12-28 11:38:30,157 INFO datanode.DataBlockScanner > (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for > blk_1323251621379_178399 > 2011-12-28 11:38:37,549 INFO DataNode.clienttrace > (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: / > 107.252.175.3:51942, bytes: 396288, op: HDFS_READ, cliID: > DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27, > srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid: > blk_1323251634345_187432 > 2011-12-28 11:38:37,550 WARN datanode.DataNode > (DataXceiver.java:readBlock(274)) - DatanodeRegistration( > 107.252.175.3:10010, > storageID=DS-306564179-107.252.175.3-10010-1322019943818, infoPort=10075, > ipcPort=10020):Got exception while serving blk_1323251634345_187432 to / > 107.252.175.3: > java.net.SocketTimeoutException: 480000 millis timeout while waiting for > channel to be ready for write. ch : > java.nio.channels.SocketChannel[connected local=/107.252.175.3:10010remote=/ > 107.252.175.3:51942] > at > org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:249) > at > org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:410) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:508) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:247) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:130) > at java.lang.Thread.run(Thread.java:662) > > Regards, > Uma > --20cf307f3714fedb5304b5cbcb1e--