Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 993C911848 for ; Thu, 24 Jul 2014 06:49:54 +0000 (UTC) Received: (qmail 32396 invoked by uid 500); 24 Jul 2014 06:49:52 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 32323 invoked by uid 500); 24 Jul 2014 06:49:52 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 32312 invoked by uid 99); 24 Jul 2014 06:49:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2014 06:49:51 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of shankar.hiremath@huawei.com designates 119.145.14.66 as permitted sender) Received: from [119.145.14.66] (HELO szxga03-in.huawei.com) (119.145.14.66) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2014 06:49:48 +0000 Received: from 172.24.2.119 (EHLO nkgeml406-hub.china.huawei.com) ([172.24.2.119]) by szxrg03-dlp.huawei.com (MOS 4.4.3-GA FastPath queued) with ESMTP id ASA34875; Thu, 24 Jul 2014 14:49:20 +0800 (CST) Received: from nkgeml510-mbs.china.huawei.com ([169.254.4.246]) by nkgeml406-hub.china.huawei.com ([10.98.56.37]) with mapi id 14.03.0158.001; Thu, 24 Jul 2014 14:49:18 +0800 From: Shankar hiremath To: "user@hbase.apache.org" Subject: RegionServer many socket fds are in CLOSE_WAIT and not getting cleared Thread-Topic: RegionServer many socket fds are in CLOSE_WAIT and not getting cleared Thread-Index: Ac+nC2JFL/qU562eQ5GTCaOfYgC/vQ== Date: Thu, 24 Jul 2014 06:49:18 +0000 Message-ID: Accept-Language: en-IN, zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.147.75] Content-Type: multipart/alternative; boundary="_000_CB0197544979BE458C310023C8F4505B091B81BAnkgeml510mbschi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020205.53D0AC71.007A,ss=1,re=0.000,fgs=0, ip=169.254.4.246, so=2013-05-26 15:14:31, dmn=2011-05-27 18:58:46 X-Mirapoint-Loop-Id: 7bc89097e7473d4217c266b75d4e29c4 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CB0197544979BE458C310023C8F4505B091B81BAnkgeml510mbschi_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Dear All, Observation: I have HBase cluster with Kerberos enabled, when the Region Server startups= then we observed some 20-30 socket fds are in CLOSE_WAIT state, After that when the Region Server starts opening then the socket fds in CLO= SE_WAIT starts increasing gradually (almost same as number of regions opene= d by the region server) And all these CLOSE_WAIT states are not getting cleared up, /hbase> lsof -i | grep `jps | grep RegionServer | cut -d " " -f1` java 18028 shankar1 118u IPv6 18552894 0t0 TCP XX-XX-XX-XX:60020= (LISTEN) java 18028 shankar1 160u IPv6 18548520 0t0 TCP *:60030 (LISTEN) java 18028 shankar1 167u IPv6 18548522 0t0 TCP XX-XX-XX-XX:42534= -> XX-XX-XX-XX:eforward (ESTABLISHED) java 18028 shankar1 172u IPv6 18552916 0t0 TCP XX-XX-XX-XX:42535= -> XX-XX-XX-XX:eforward (ESTABLISHED) java 18028 shankar1 173u IPv6 18551227 0t0 TCP XX-XX-XX-XX:49646= ->XX-XX-XX-XX:60000 (ESTABLISHED) java 18028 shankar1 178u IPv6 18551237 0t0 TCP XX-XX-XX-XX:62668= ->XX-XX-XX-XX:busboy (ESTABLISHED) java 18028 shankar1 185u IPv6 18548549 0t0 TCP XX-XX-XX-XX:21856= -> XX-XX-XX-XX:eforward (ESTABLISHED) java 18028 shankar1 187u IPv6 18548558 0t0 TCP XX-XX-XX-XX:62673= ->XX-XX-XX-XX:busboy (ESTABLISHED) java 18028 shankar1 188u IPv6 18601323 0t0 TCP XX-XX-XX-XX:63168= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 189u IPv6 18601322 0t0 TCP XX-XX-XX-XX:63167= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 190u IPv6 18601324 0t0 TCP XX-XX-XX-XX:63169= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 191r IPv6 18592423 0t0 TCP XX-XX-XX-XX:63087= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 193u IPv6 18593210 0t0 TCP XX-XX-XX-XX:63090= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 194u IPv6 18548560 0t0 TCP XX-XX-XX-XX:62675= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 195u IPv6 18592428 0t0 TCP XX-XX-XX-XX:63093= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 196u IPv6 18593218 0t0 TCP XX-XX-XX-XX:63096= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 197u IPv6 18591423 0t0 TCP XX-XX-XX-XX:63105= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 201u IPv6 18592431 0t0 TCP XX-XX-XX-XX:63099= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 202u IPv6 18592433 0t0 TCP XX-XX-XX-XX:63102= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) java 18028 shankar1 203u IPv6 18552317 0t0 TCP XX-XX-XX-XX:62681= ->XX-XX-XX-XX:busboy (CLOSE_WAIT) ...........................................................................= ................................. ...........................................................................= ................................. ...........................................................................= ................................. ...........................................................................= ................................. ...........................................................................= ................................. any input or suggestion will be helpful, or is it a bug. Regards -Shankar [X] This e-mail and its attachments contain confidential information from HUAWE= I, which is intended only for the person or entity whose address is listed = above. Any use of the information contained herein in any way (including, b= ut not limited to, total or partial disclosure, reproduction, or disseminat= ion) by persons other than the intended recipient(s) is prohibited. If you = receive this e-mail in error, please notify the sender by phone or email im= mediately and delete it! [X] --_000_CB0197544979BE458C310023C8F4505B091B81BAnkgeml510mbschi_--