Mailing-List: contact dev-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flume.apache.org
Date: Thu, 12 May 2016 19:57:13 +0000 (UTC)
From: "Jarek Jarcec Cecho (JIRA)" <jira@apache.org>
To: dev@flume.apache.org
Message-ID: <JIRA.12964178.1462145735000.179527.1463083033150@Atlassian.JIRA>
In-Reply-To: <JIRA.12964178.1462145735000@Atlassian.JIRA>
References: <JIRA.12964178.1462145735000@Atlassian.JIRA> <JIRA.12964178.1462145735020@arcas>
Subject: [jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed
 when an exception is encountered during start() leading to file descriptor
 leaks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 12 May 2016 19:57:20 -0000


    [ https://issues.apache.org/jira/browse/FLUME-2905?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1528=
1953#comment-15281953 ]=20

Jarek Jarcec Cecho commented on FLUME-2905:
-------------------------------------------

Sure, my pleasure.  Quickly looking at the patch, can you please remove the=
 trailing white space warnings [~sahuja]?

{code}
jarcec@arlene flume % git apply FLUME-2905-0.patch                         =
                                                                           =
                                                                           =
                                                                           =
               [trunk =E2=9C=97] (1.8.0_77-b03) (ruby-2.2.3) [12:54:00]
FLUME-2905-0.patch:25: space before tab in indent.
    =09.setNameFormat("netcat-handler-%d").build());
FLUME-2905-0.patch:50: trailing whitespace.

FLUME-2905-0.patch:52: trailing whitespace.
   * Tests that the source is stopped when an exception is thrown
FLUME-2905-0.patch:54: trailing whitespace.
   * clean up the sockets opened during source.start().
FLUME-2905-0.patch:55: trailing whitespace.
   *
warning: squelched 5 whitespace errors
warning: 10 lines add whitespace errors.
{code}

> NetcatSource - Socket not closed when an exception is encountered during =
start() leading to file descriptor leaks
> -------------------------------------------------------------------------=
----------------------------------------
>
>                 Key: FLUME-2905
>                 URL: https://issues.apache.org/jira/browse/FLUME-2905
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.6.0
>            Reporter: Siddharth Ahuja
>            Assignee: Siddharth Ahuja
>         Attachments: FLUME-2905-0.patch
>
>
> During the flume agent start-up, the flume configuration containing the N=
etcatSource is parsed and the source's start() is called. If there is an is=
sue while binding the channel's socket to a local address to configure the =
socket to listen for connections following exception is thrown but the sock=
et open just before is not closed.=20
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervi=
sor: Unable to start EventDrivenSourceRunner: { source:org.apache.flume.sou=
rce.NetcatSource{name:src-1,state:IDLE} } - Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already =
in use
>         at org.apache.flume.source.NetcatSource.start(NetcatSource.java:1=
73)
>         at org.apache.flume.source.EventDrivenSourceRunner.start(EventDri=
venSourceRunner.java:44)
>         at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable=
.run(LifecycleSupervisor.java:251)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.=
java:471)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:30=
4)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu=
reTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu=
reTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx=
ecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
>         at sun.nio.ch.Net.bind0(Native Method)
>         at sun.nio.ch.Net.bind(Net.java:444)
>         at sun.nio.ch.Net.bind(Net.java:436)
>         at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImp=
l.java:214)
>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:7=
4)
>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:6=
7)
>         at org.apache.flume.source.NetcatSource.start(NetcatSource.java:1=
67)
>         ... 9 more
> {code}
> The source's start() is then called again leading to another socket being=
 opened but not closed and so on. This leads to file descriptor (socket) le=
aks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in =
use. e.g. in my case I used 50010 which is the port for DataNode's XCeiver =
Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p <flume_process_id> | wc -l". No=
tice the file descriptors keep on growing due to socket leaks with errors l=
ike: "can't identify protocol".


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)