Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A79E910767 for ; Tue, 22 Oct 2013 07:03:20 +0000 (UTC) Received: (qmail 76381 invoked by uid 500); 22 Oct 2013 07:03:02 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 75668 invoked by uid 500); 22 Oct 2013 07:02:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 75648 invoked by uid 99); 22 Oct 2013 07:02:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Oct 2013 07:02:43 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vkjk89@gmail.com designates 209.85.223.180 as permitted sender) Received: from [209.85.223.180] (HELO mail-ie0-f180.google.com) (209.85.223.180) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Oct 2013 07:02:37 +0000 Received: by mail-ie0-f180.google.com with SMTP id e14so984985iej.11 for ; Tue, 22 Oct 2013 00:02:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=33LxAqKePhr/foaPPNUML7ottHZUP2bxJrpGeY8WEcA=; b=DVDFvNa2BB6B5JKzZs+FuwclKSV+8xmPeuYfFZz5Kg6RlGEf/1RB0V2RqmZE0otEwW k2Z87VI8yD8FlH17dV6eEiN2/IU4NfuoTvEFrS1KDh4njKP6zOitqWQUWeu6mKhYkuQw l6cAPT73gG685r8QZmIsZm08AR3xCBpVrR1w2WnbyBv/Ofw7IcpV3KQL3/WwTKhVnAJF IgAM6KuGUHdL1QnIuZxrvrsDb6a/RHoJBLsY3LWmUbg78LqCDNMY71BPJ4wjluAcmYVH PxBQ2Q8vt5yhfNwDUOuo7VK5VaRi40Z7WIA2ar4puqXLuHdeODe2545jqMdriJzLbMok A0Sw== MIME-Version: 1.0 X-Received: by 10.43.60.139 with SMTP id ws11mr1287919icb.12.1382425336117; Tue, 22 Oct 2013 00:02:16 -0700 (PDT) Received: by 10.64.9.237 with HTTP; Tue, 22 Oct 2013 00:02:16 -0700 (PDT) Date: Tue, 22 Oct 2013 12:32:16 +0530 Message-ID: Subject: High Full GC count for Region server From: Vimal Jain To: "user@hbase.apache.org" , user@hadoop.apache.org Content-Type: multipart/alternative; boundary=bcaec51a89461680ff04e94efb69 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51a89461680ff04e94efb69 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, I am running in Hbase in pseudo distributed mode. ( Hadoop version - 1.1.2 , Hbase version - 0.94.7 ) I am getting few exceptions in both hadoop ( namenode , datanode) logs and hbase(region server). When i search for these exceptions on google , i concluded that problem is mainly due to large number of full GC in region server process. I used jstat and found that there are total of 950 full GCs in span of 4 days for region server process.Is this ok? I am totally confused by number of exceptions i am getting. Also i get below exceptions intermittently. Region server:- 2013-10-22 12:00:26,627 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":15312,"call":"next(-6681408251916104762, 1000), rpc version=3D1, client version=3D29, methodsFingerPrint=3D-1368823753","client= ":" 192.168.20.31:48270 ","starttimems":1382423411293,"queuetimems":0,"class":"HRegionServer","resp= onsesize":4808556,"method":"next"} 2013-10-22 12:06:17,606 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":14759,"client":"192.168.20.31:48247 ","timeRange":[0,9223372036854775807],"starttimems":1382423762845,"response= size":61,"class":"HRegionServer","table":"event_data","cacheBlocks":true,"f= amilies":{"ginfo":["netGainPool"]},"row":"1629657","queuetimems":0,"method"= :"get","totalColumns":1,"maxVersions":1} 2013-10-18 10:37:45,008 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/event_data/4c3765c51911d6c67037a983d205a010/.tmp/bfaf8df33d5b4068825= e3664d3e4b2b0 could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNa= mesystem.java:1639) Name node :- java.io.IOException: File /hbase/event_data/433b61f2a4ebff8f2e4b89890508a3b7/.tmp/99797a61a8f7471cb6d= f8f7b95f18e9e could only be replicated to 0 nodes, instead of 1 java.io.IOException: Got blockReceived message from unregistered or dead node blk_-2949905629769882833_52274 Data node :- 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=3D/192.168.20.30:50010remot= e=3D/ 192.168.20.30:36188] ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration= ( 192.168.20.30:50010, storageID=3DDS-1816106352-192.168.20.30-50010-1369314076237, infoPort=3D500= 75, ipcPort=3D50020):DataXceiver java.io.EOFException: while trying to read 39309 bytes --=20 Thanks and Regards, Vimal Jain --bcaec51a89461680ff04e94efb69 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,
I am running in Hbas= e in pseudo distributed mode. ( Hadoop version - 1.1.2 , Hbase version - 0.= 94.7 )
I am getting few exceptions in both hadoop ( namenode , dat= anode) logs and hbase(region server).
When i search for these exceptions on google , i concluded=A0 that pr= oblem is mainly due to large number of full GC in region server process.
I used jstat and found that there are total of 950 full GCs in s= pan of 4 days for region server process.Is this ok?

I am totally confused by number of exceptions i am getting.<= br>Also i get below exceptions intermittently.


Region= server:-

2013-10-22 12:00:26,627 WARN org.apache.hadoop.ipc.HBaseSe= rver: (responseTooSlow): {"processingtimems":15312,"call&quo= t;:"next(-6681408251916104762, 1000), rpc version=3D1, client version= =3D29, methodsFingerPrint=3D-1368823753","client":"192.168.20.31:48270","star= ttimems":1382423411293,"queuetimems":0,"class":&qu= ot;HRegionServer","responsesize":4808556,"method":= "next"}
2013-10-22 12:06:17,606 WARN org.apache.hadoop.ipc.HBaseServer: (operationT= ooSlow): {"processingtimems":14759,"client":"192.168.20.31:48247","timeR= ange":[0,9223372036854775807],"starttimems":1382423762845,&q= uot;responsesize":61,"class":"HRegionServer","= ;table":"event_data","cacheBlocks":true,"fami= lies":{"ginfo":["netGainPool"]},"row":&q= uot;1629657","queuetimems":0,"method":"get&qu= ot;,"totalColumns":1,"maxVersions":1}

2013-10-18 10:37:45,008 WARN org.apache.hadoop.hdfs.DFSClient: DataStre= amer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException:= File /hbase/event_data/4c3765c51911d6c67037a983d205a010/.tmp/bfaf8df33d5b4= 068825e3664d3e4b2b0 could only be replicated to 0 nodes, instead of 1
=A0=A0=A0 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditio= nalBlock(FSNamesystem.java:1639)

Name node :-
java.io.= IOException: File /hbase/event_data/433b61f2a4ebff8f2e4b89890508a3b7/.tmp/9= 9797a61a8f7471cb6df8f7b95f18e9e could only be replicated to 0 nodes, instea= d of 1

java.io.IOException: Got blockReceived message from unregistered or dea= d node blk_-2949905629769882833_52274

Data node :-
480= 000 millis timeout while waiting for channel to be ready for write. ch : ja= va.nio.channels.SocketChannel[connected local=3D/192.168.20.30:50010 remote=3D/192.168.20.30:36188]

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistra= tion(192.168.20.30:50010, storag= eID=3DDS-1816106352-192.168.20.30-50010-1369314076237, infoPort=3D50075, ip= cPort=3D50020):DataXceiver
java.io.EOFException: while trying to read 39309 bytes


--
Thanks and Regards,
Vimal Jain
--bcaec51a89461680ff04e94efb69--