nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Kelly (Jira)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-7114) NiFi not closing file handles
Date Fri, 21 Feb 2020 12:13:00 GMT

    [ https://issues.apache.org/jira/browse/NIFI-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041791#comment-17041791
] 

Paul Kelly commented on NIFI-7114:
----------------------------------

So my overnight test didn't leak, but I was able to find something that does leak FIFOs 100%
of the time.  Please see the new flow.xml.gz that is attached.  It is much simpler: GenerateFlowFile
-> DetectDuplicate (with a DistributedMapCache as its backing service) -> LogAttribute.

Our normal data flow is pretty bursty.  We don't have a continuous stream of data and we
sometimes go a few minutes without anything.  So to simulate this, I kept a repeated watch
of "lsof | grep (java pid) | grep FIFO | wc -l" running while stopping the GenerateFlowFile
processor for one minute, then turning it back on, and repeating.  The effects can be seen
in the attached fifocounts.txt.

What happens is immediately after stopping GenerateFlowFile, the FIFO counts hold steady for
around one minute.  After that minute elapses (when some sort of garbage collection runs?),
they drop by ~150.  Immediately after restarting GenerateFlowFile, they jump up by ~300 and
stay there.  I let it run for an hour after and it doesn't seem like they ever get closed. 
These effects are reproducible 100% of the time.

It seems like whatever is causing the FIFOs to be opened opens two of them when they start,
but only 1 is getting closed when the flow is idle.

> NiFi not closing file handles
> -----------------------------
>
>                 Key: NIFI-7114
>                 URL: https://issues.apache.org/jira/browse/NIFI-7114
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Configuration
>    Affects Versions: 1.10.0, 1.11.0
>         Environment: Amazon EC2 running either Amazon Linux 2 or Ubuntu 18.04.
> NiFi has been installed with no change to any configuration file.
>            Reporter: Vinicius Zolin
>            Priority: Major
>         Attachments: destination.xml, fifocounts.txt, flow.xml.gz, lsof.log, lsof.zip,
lsofAfter.log, lsofBefore.log, openFiles.xlsx, reproduction.zip, source.xml
>
>
> Since at least version 1.10 NiFi stopped closing file handles. It opens circa 500 files
per hour (measured using lsof) without any apparent limit until it crashes due to too many
open files.
>  
> Increasing the computer open file limit is not a solution since NiFi will still crash,
it'll only take longer to do so.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message