nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Kelly (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (NIFI-7114) NiFi not closing file handles
Date Fri, 21 Feb 2020 15:54:00 GMT

    [ https://issues.apache.org/jira/browse/NIFI-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041973#comment-17041973
] 

Paul Kelly edited comment on NIFI-7114 at 2/21/20 3:53 PM:
-----------------------------------------------------------

It looks like lsof -p (pid) is growing for me as well.  I adjusted the scheduling on the
GenerateFlowFile to generate 200 files every 45 seconds and changed the size to 100B.  Now
I don't have to start and stop the processor to see the batching effect.  I've only had it
running for about 15 minutes and already the FIFOs within LSOF -p have grown from 21 to 39. 
I'll keep it running.  This isn't nearly as dramatic as the growth of FIFOs in our production
flow, but it is happening.

I upgraded our production environment to 1.11.2 again just now and have it running without
restarts.  I will try to grab some logs when it crashes.

I agree with [~vzolin] that it seems to be related to our use of DistributedMapCache.  Setting
up a new flow using one on 1.10.0 (on Windows) involving a DistributedMapCache was the first
time I experienced an running out of available TCP client sockets at the OS level with NiFi. 
After upgrading our Linux servers to 1.10.0 a week or so later, we ran into the file descriptor
issue with FIFOs.  From some research, it seems that java.nio uses socket pairs on Windows
and pipes on Unix for IPC.  So a leak with some java.nio IPC would cause what we are seeing
on both OSes.


was (Author: pkelly.nifi):
It looks like lsof -p (pid) is growing for me as well.  I adjusted the scheduling on the
GenerateFlowFile to generate 200 files every 45 seconds and changed the size to 100B.  Now
I don't have to start and stop the processor to see the batching effect.  I've only had it
running for about 15 minutes and already the FIFOs within LSOF -p have grown from 21 to 39. 
I'll keep it running.  This isn't nearly as dramatic as our production flow, but it is happening.

I upgraded our production environment to 1.11.2 again just now and have it running without
restarts.  I will try to grab some logs when it crashes.

I agree with [~vzolin] that it seems to be related to our use of DistributedMapCache.  Setting
up a new flow using one on 1.10.0 (on Windows) involving a DistributedMapCache was the first
time I experienced an running out of available TCP client sockets at the OS level with NiFi. 
After upgrading our Linux servers to 1.10.0 a week or so later, we ran the file descriptor
issue with FIFOs.  From some research, it seems that java.nio uses socket pairs on Windows
and pipes on Unix for IPC.  So a leak with some java.nio IPC would cause what we are seeing
on both OSes.

> NiFi not closing file handles
> -----------------------------
>
>                 Key: NIFI-7114
>                 URL: https://issues.apache.org/jira/browse/NIFI-7114
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Configuration
>    Affects Versions: 1.10.0, 1.11.0
>         Environment: Amazon EC2 running either Amazon Linux 2 or Ubuntu 18.04.
> NiFi has been installed with no change to any configuration file.
>            Reporter: Vinicius Zolin
>            Priority: Major
>         Attachments: destination.xml, fifocounts.txt, flow.xml.gz, lsof.log, lsof.zip,
lsofAfter.log, lsofBefore.log, openFiles.xlsx, reproduction.zip, source.xml
>
>
> Since at least version 1.10 NiFi stopped closing file handles. It opens circa 500 files
per hour (measured using lsof) without any apparent limit until it crashes due to too many
open files.
>  
> Increasing the computer open file limit is not a solution since NiFi will still crash,
it'll only take longer to do so.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message