Return-Path: <issues-return-34136-archive-asf-public=cust-asf.ponee.io@mesos.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id A4C61200BA3
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Thu, 20 Oct 2016 10:40:00 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id A358A160AFE; Thu, 20 Oct 2016 08:40:00 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id CB048160ADB
	for <archive-asf-public@cust-asf.ponee.io>; Thu, 20 Oct 2016 10:39:59 +0200 (CEST)
Received: (qmail 90193 invoked by uid 500); 20 Oct 2016 08:39:58 -0000
Mailing-List: contact issues-help@mesos.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@mesos.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@mesos.apache.org>
List-Post: <mailto:issues@mesos.apache.org>
List-Id: <issues.mesos.apache.org>
Reply-To: dev@mesos.apache.org
Delivered-To: mailing list issues@mesos.apache.org
Received: (qmail 90178 invoked by uid 99); 20 Oct 2016 08:39:58 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2016 08:39:58 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 8D1D32C0D53
	for <issues@mesos.apache.org>; Thu, 20 Oct 2016 08:39:58 +0000 (UTC)
Date: Thu, 20 Oct 2016 08:39:58 +0000 (UTC)
From: "Santhosh Shanmugham (JIRA)" <jira@apache.org>
To: issues@mesos.apache.org
Message-ID: <JIRA.13013658.1476910813000.37656.1476952798574@Atlassian.JIRA>
In-Reply-To: <JIRA.13013658.1476910813000@Atlassian.JIRA>
References: <JIRA.13013658.1476910813000@Atlassian.JIRA> <JIRA.13013658.1476910813359@arcas>
Subject: [jira] [Comment Edited] (MESOS-6420) Mesos Agent leaking sockets
 when port mapping network isolator is ON
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Thu, 20 Oct 2016 08:40:00 -0000


    [ https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15591210#comment-15591210 ] 

Santhosh Shanmugham edited comment on MESOS-6420 at 10/20/16 8:39 AM:
----------------------------------------------------------------------

The socker appears to be leaked inside {{src/linux/routing/link/link.cpp}}. See line 160 in {{src/linux/routing/link/link.cpp}}

{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try<bool> enable = link::setUp(veth(pid));
{code}

{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try<bool> setUp(const string& link)
244 {
245   return internal::set(link, IFF_UP);
246 }
{code}

{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try<bool> set(const std::string& _link, unsigned int flags)
126 {
...
144   int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145   if (fd == -1) {
146     return ErrnoError();
147   }
148
149   if (ioctl(fd, SIOCSIFFLAGS, &ifr) == -1) {
150     if (errno == ENODEV) {
151       os::close(fd);
152       return false;
153     } else {
154       // Save the error string as os::close may overwrite errno.
155       const std::string message = os::strerror(errno);
156       os::close(fd);
157       return Error(message);
158     }
159   }
160   // os::close(fd);  <<==== MISSING
161   return true;
162 }
{code}


was (Author: santhoshkumar.s@gmail.com):
The socker appears to be leaked inside {{src/linux/routing/link/link.cpp}}.

{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try<bool> enable = link::setUp(veth(pid));
{code}

{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try<bool> setUp(const string& link)
244 {
245   return internal::set(link, IFF_UP);
246 }
{code}

{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try<bool> set(const std::string& _link, unsigned int flags)
126 {
...
144   int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145   if (fd == -1) {
146     return ErrnoError();
147   }
148
149   if (ioctl(fd, SIOCSIFFLAGS, &ifr) == -1) {
150     if (errno == ENODEV) {
151       os::close(fd);
152       return false;
153     } else {
154       // Save the error string as os::close may overwrite errno.
155       const std::string message = os::strerror(errno);
156       os::close(fd);
157       return Error(message);
158     }
159   }
160   // os::close(fd);  <<==== MISSING
161   return true;
162 }
{code}

> Mesos Agent leaking sockets when port mapping network isolator is ON
> --------------------------------------------------------------------
>
>                 Key: MESOS-6420
>                 URL: https://issues.apache.org/jira/browse/MESOS-6420
>             Project: Mesos
>          Issue Type: Bug
>          Components: isolation, network, slave
>    Affects Versions: 1.0.2
>            Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of sockets. We were able to track it down to the network isolator (port_mapping.cpp). When we turned off the port mapping isolator no file descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> $ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root   19u  sock        0,6      0t0 2993216948 can't identify protocol
> mesos-sla 57688 root   27u  sock        0,6      0t0 2993216468 can't identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19)   = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19)   = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19)   = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19)   = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19)   = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19)   = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19)   = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19)   = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19)   = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18 <unfinished ...>
> [pid 57701] 19:14:02.499634 close(19)   = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19)   = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19)   = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.501271 close(19)   = 0
> [pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.502030 close(19)   = 0
> [pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
> ...
> ...
> [pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid  6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY <unfinished ...>
> [pid 57691] 19:18:03.461460 close(27)   = 0
> [pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid  6138] 19:18:03.461632 close(3 <unfinished ...>
> [pid  6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY <unfinished ...>
> [pid  6138] 19:18:03.462190 close(3 <unfinished ...>
> [pid 57691] 19:18:03.462374 close(27)   = 0
> [pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0 <unfinished ...>
> [pid  6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY <unfinished ...>
> [pid  6138] 19:18:03.462678 close(3 <unfinished ...>
> [pid  6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY <unfinished ...>
> [pid 57691] 19:18:03.463046 close(27)   = 0
> [pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid  6138] 19:18:03.463225 close(3 <unfinished ...>
> [pid 57691] 19:18:03.463845 close(27)   = 0
> [pid 57691] 19:18:03.463911 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.464604 close(27)   = 0
> [pid 57691] 19:18:03.464664 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465074 close(27)   = 0
> [pid 57691] 19:18:03.465132 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465862 close(27)   = 0
> [pid 57691] 19:18:03.465928 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.466713 close(27)   = 0
> [pid 57691] 19:18:03.466780 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.467472 close(27)   = 0
> [pid 57691] 19:18:03.467524 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.468012 close(27)   = 0
> [pid 57691] 19:18:03.468075 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.468799 close(27)   = 0
> [pid 57691] 19:18:03.468950 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.469505 close(27)   = 0
> [pid 57691] 19:18:03.469578 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.470301 close(27)   = 0
> [pid 57691] 19:18:03.470353 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 27
> ...
> The last socket the was created never has a corresponding close().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)