From issues-return-92575-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Fri Feb 21 15:54:03 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 60849180674 for ; Fri, 21 Feb 2020 16:54:03 +0100 (CET) Received: (qmail 84019 invoked by uid 500); 21 Feb 2020 15:54:02 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 83926 invoked by uid 99); 21 Feb 2020 15:54:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Feb 2020 15:54:02 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id A7EC9E3163 for ; Fri, 21 Feb 2020 15:54:01 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 6286C7808F1 for ; Fri, 21 Feb 2020 15:54:00 +0000 (UTC) Date: Fri, 21 Feb 2020 15:54:00 +0000 (UTC) From: "Paul Kelly (Jira)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (NIFI-7114) NiFi not closing file handles MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/NIFI-7114?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D17041= 973#comment-17041973 ]=20 Paul Kelly edited comment on NIFI-7114 at 2/21/20 3:53 PM: ----------------------------------------------------------- It looks like lsof -p (pid) is growing for me as well.=C2=A0 I adjusted the= scheduling on the GenerateFlowFile to generate 200 files every 45 seconds = and changed the size to 100B.=C2=A0 Now I don't have to start and stop the = processor to see the batching effect.=C2=A0 I've only had it running for ab= out 15 minutes and already the FIFOs within LSOF -p have grown from 21 to 3= 9.=C2=A0 I'll keep it running.=C2=A0 This isn't nearly as dramatic as the g= rowth of FIFOs in our production flow, but it is happening. I upgraded our production environment to 1.11.2 again just now and have it = running without restarts.=C2=A0 I will try to grab some logs when it crashe= s. I agree with [~vzolin] that it seems to be related to our use of Distribute= dMapCache.=C2=A0 Setting up a new flow using one on 1.10.0 (on Windows) inv= olving a DistributedMapCache was the first time I experienced an running ou= t of available TCP client sockets at the OS level with NiFi.=C2=A0 After up= grading our Linux servers to 1.10.0 a week or so later, we ran into the fil= e descriptor issue with FIFOs.=C2=A0 From some research, it seems that java= .nio uses socket pairs on Windows and pipes on Unix for IPC.=C2=A0 So a lea= k with some java.nio IPC would cause what we are seeing on both OSes. was (Author: pkelly.nifi): It looks like lsof -p (pid) is growing for me as well.=C2=A0 I adjusted the= scheduling on the GenerateFlowFile to generate 200 files every 45 seconds = and changed the size to 100B.=C2=A0 Now I don't have to start and stop the = processor to see the batching effect.=C2=A0 I've only had it running for ab= out 15 minutes and already the FIFOs within LSOF -p have grown from 21 to 3= 9.=C2=A0 I'll keep it running.=C2=A0 This isn't nearly as dramatic as our p= roduction flow, but it is happening. I upgraded our production environment to 1.11.2 again just now and have it = running without restarts.=C2=A0 I will try to grab some logs when it crashe= s. I agree with [~vzolin] that it seems to be related to our use of Distribute= dMapCache.=C2=A0 Setting up a new flow using one on 1.10.0 (on Windows) inv= olving a DistributedMapCache was the first time I experienced an running ou= t of available TCP client sockets at the OS level with NiFi.=C2=A0 After up= grading our Linux servers to 1.10.0 a week or so later, we ran the file des= criptor issue with FIFOs.=C2=A0 From some research, it seems that java.nio = uses socket pairs on Windows and pipes on Unix for IPC.=C2=A0 So a leak wit= h some java.nio IPC would cause what we are seeing on both OSes. > NiFi not closing file handles > ----------------------------- > > Key: NIFI-7114 > URL: https://issues.apache.org/jira/browse/NIFI-7114 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration > Affects Versions: 1.10.0, 1.11.0 > Environment: Amazon EC2 running either Amazon Linux 2 or Ubuntu 1= 8.04. > NiFi has been installed with no change to any configuration file. > Reporter: Vinicius Zolin > Priority: Major > Attachments: destination.xml, fifocounts.txt, flow.xml.gz, lsof.l= og, lsof.zip, lsofAfter.log, lsofBefore.log, openFiles.xlsx, reproduction.z= ip, source.xml > > > Since at least version 1.10 NiFi stopped closing file handles. It opens c= irca 500 files per hour (measured using lsof) without any apparent limit un= til it crashes due to too many open files. > =C2=A0 > Increasing the computer open file limit is not a solution since NiFi will= still crash, it'll only take longer to do so. -- This message was sent by Atlassian Jira (v8.3.4#803005)