From user-return-7219-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org Thu Jan 25 18:23:24 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id A7D94180651 for ; Thu, 25 Jan 2018 18:23:24 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 97212160C3D; Thu, 25 Jan 2018 17:23:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B5899160C13 for ; Thu, 25 Jan 2018 18:23:23 +0100 (CET) Received: (qmail 47674 invoked by uid 500); 25 Jan 2018 17:23:22 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 47665 invoked by uid 99); 25 Jan 2018 17:23:22 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jan 2018 17:23:22 +0000 Received: from mail-vk0-f42.google.com (mail-vk0-f42.google.com [209.85.213.42]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 6EF531A0044 for ; Thu, 25 Jan 2018 17:23:22 +0000 (UTC) Received: by mail-vk0-f42.google.com with SMTP id g83so5307778vki.4 for ; Thu, 25 Jan 2018 09:23:22 -0800 (PST) X-Gm-Message-State: AKwxytcBEaqAH/SlCRQJnJJSQfyHbcJG6AmNrrD7oLwGIRv4WGx4vzHN r7NEgV31JbBpwqKFz02N3W/OViEVHYdjeyOwvb8= X-Google-Smtp-Source: AH8x225MqT2Jm423+uuz69XkmVwcwWvmqtniNBMAhFpNbaS88rcBAasZgw6vqFl3/VqDsRsbvVmJK1j+4poHClhwSPE= X-Received: by 10.31.3.94 with SMTP id 91mr7992826vkd.124.1516901001407; Thu, 25 Jan 2018 09:23:21 -0800 (PST) MIME-Version: 1.0 References: <2f6596b6-122f-e1c5-4780-c9eca0fd68fc@gmail.com> In-Reply-To: From: Christopher Date: Thu, 25 Jan 2018 17:23:10 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Large number of used ports from tserver To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary="001a1142873c5932da05639d0933" --001a1142873c5932da05639d0933 Content-Type: text/plain; charset="UTF-8" Interesting. It's possible we're mishandling an IOException from DFSClient or something... but it's also possible there's a bug in DFSClient somewhere. I found a few similar issues from the past... some might still be not fully resolved: https://issues.apache.org/jira/browse/HDFS-1836 https://issues.apache.org/jira/browse/HDFS-2028 https://issues.apache.org/jira/browse/HDFS-6973 https://issues.apache.org/jira/browse/HBASE-9393 The HBASE issue is interesting, because it indicates a new HDFS feature in 2.6.4 to clear readahead buffers/sockets ( https://issues.apache.org/jira/browse/HDFS-7694). That might be a feature we're not yet utilizing, but it would only work on a newer version of HDFS. I would probably also try to grab some jstacks of the tserver, to try to figure out what HDFS client code paths are being taken to see where the leak might be occurring. Also, if you have any debug logs for the tserver, that might help. There might be some DEBUG or WARN items that indicate retries or other failures failures that are occurring, but perhaps handled improperly. It's probably less likely, but it could also be a Java or Linux issue. I wouldn't even know where to begin debugging at that level, though, other than to check for OS updates. What JVM are you running? It's possible it's not a leak... and these are just getting cleaned up too slowly. That might be something that can be tuned with sysctl. On Thu, Jan 25, 2018 at 11:27 AM Adam J. Shook wrote: > We're running Ubuntu 14.04, HDFS 2.6.0, ZooKeeper 3.4.6, and Accumulo > 1.8.1. I'm using `lsof -i` and grepping for the tserver PID to list all > the connections. Just now there are ~25k connections for this one tserver, > of which 99.9% of them are all writing to various DataNodes on port 50010. > It's split about 50/50 for connections that are CLOSED_WAIT and ones that > are ESTABLISHED. No special RPC configuration. > > On Wed, Jan 24, 2018 at 7:53 PM, Josh Elser wrote: > >> +1 to looking at the remote end of the socket and see where they're >> going/coming to/from. I've seen a few HDFS JIRA issues filed about sockets >> left in CLOSED_WAIT. >> >> Lucky you, this is a fun Linux rabbit hole to go down :) >> >> ( >> https://blog.cloudflare.com/this-is-strictly-a-violation-of-the-tcp-specification/ >> covers some of the technical details) >> >> On 1/24/18 6:37 PM, Christopher wrote: >> >>> I haven't seen that, but I'm curious what OS, Hadoop, ZooKeeper, and >>> Accumulo version you're running. I'm assuming you verified that it was the >>> TabletServer process holding these TCP sockets open using `netstat -p` and >>> cross-referencing the PID with `jps -ml` (or similar)? Are you able to >>> confirm based on the port number that these were Thrift connections or >>> could they be ZooKeeper or Hadoop connections? Do you have any special >>> non-default Accumulo RPC configuration (SSL or SASL)? >>> >>> On Wed, Jan 24, 2018 at 3:46 PM Adam J. Shook >> > wrote: >>> >>> Hello all, >>> >>> Has anyone come across an issue with a TabletServer occupying a >>> large number of ports in a CLOSED_WAIT state? 'Normal' number of >>> used ports on a 12-node cluster are around 12,000 to 20,000 ports. >>> In one instance, there were over 68k and it was affecting other >>> applications from getting a free port and they would fail to start >>> (which is how we found this in the first place). >>> >>> Thank you, >>> --Adam >>> >>> > --001a1142873c5932da05639d0933 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Interesting. It's possible we're mishandling an IO= Exception from DFSClient or something... but it's also possible there&#= 39;s a bug in DFSClient somewhere. I found a few similar issues from the pa= st... some might still be not fully resolved:

https://issues.apache.org/jira/browse= /HDFS-1836
https://issues.apache.org/jira/browse/HDFS-2028
https://issues.apache.org/jira/b= rowse/HDFS-6973
https://issues.apache.org/jira/browse/HBASE-9393

The HB= ASE issue is interesting, because it indicates a new HDFS feature in 2.6.4 = to clear readahead buffers/sockets (https://issues.apache.org/jira/browse/HDFS-7694). = That might be a feature we're not yet utilizing, but it would only work= on a newer version of HDFS.

I would probably also try to grab some = jstacks of the tserver, to try to figure out what HDFS client code paths ar= e being taken to see where the leak might be occurring. Also, if you have a= ny debug logs for the tserver, that might help. There might be some DEBUG o= r WARN items that indicate retries or other failures failures that are occu= rring, but perhaps handled improperly.

It's probably less likely= , but it could also be a Java or Linux issue. I wouldn't even know wher= e to begin debugging at that level, though, other than to check for OS upda= tes.=C2=A0 What JVM are you running?

It's possible it's not = a leak... and these are just getting cleaned up too slowly. That might be s= omething that can be tuned with sysctl.

On Thu, Jan 25, 2018 at 11:27 AM Adam J. Shook <adamjshook@gmail.com> wrote:
We're running Ubunt= u 14.04, HDFS 2.6.0, ZooKeeper 3.4.6, and Accumulo 1.8.1.=C2=A0 I'm usi= ng `lsof -i` and grepping for the tserver PID to list all the connections.= =C2=A0 Just now there are ~25k connections for this one tserver, of which 9= 9.9% of them are all writing to various DataNodes on port 50010.=C2=A0 It&#= 39;s split about 50/50 for connections that are CLOSED_WAIT and ones that a= re ESTABLISHED.=C2=A0 No special RPC configuration.
<= div class=3D"gmail_extra">
On Wed, Jan 24, 20= 18 at 7:53 PM, Josh Elser <josh.elser@gmail.com> wrote:
+1 to looking at the remote end of the soc= ket and see where they're going/coming to/from. I've seen a few HDF= S JIRA issues filed about sockets left in CLOSED_WAIT.

Lucky you, this is a fun Linux rabbit hole to go down :)

(https://blog.clou= dflare.com/this-is-strictly-a-violation-of-the-tcp-specification/ cover= s some of the technical details)

On 1/24/18 6:37 PM, Christopher wrote:
I haven't seen that, but I'm curious what OS, Hadoop, ZooKeeper, an= d Accumulo version you're running. I'm assuming you verified that i= t was the TabletServer process holding these TCP sockets open using `netsta= t -p` and cross-referencing the PID with `jps -ml` (or similar)? Are you ab= le to confirm based on the port number that these were Thrift connections o= r could they be ZooKeeper or Hadoop connections? Do you have any special no= n-default Accumulo RPC configuration (SSL or SASL)?

 On Wed, Jan 24, 2018 at 3:46 PM Adam J. Shook <adamjshook@gmail.com <mailto:adamjshook@gmail.com>> wrote:

=C2=A0 =C2=A0 Hello all,

=C2=A0 =C2=A0 Has anyone come across an issue with a TabletServer occupying= a
=C2=A0 =C2=A0 large number of ports in a CLOSED_WAIT state?=C2=A0 'Norm= al' number of
=C2=A0 =C2=A0 used ports on a 12-node cluster are around 12,000 to 20,000 p= orts.=C2=A0 =C2=A0 =C2=A0In one instance, there were over 68k and it was af= fecting other
=C2=A0 =C2=A0 applications from getting a free port and they would fail to = start
=C2=A0 =C2=A0 (which is how we found this in the first place).

=C2=A0 =C2=A0 Thank you,
=C2=A0 =C2=A0 --Adam


--001a1142873c5932da05639d0933--