nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <>
Subject Re: FetchFile Cannot Allocate Enough Memory
Date Wed, 04 May 2016 13:06:25 GMT
ListFile performs a listing using Java's File.listFiles(). This will provide a list of all
files in the
directory. I do not believe this to be related, though. Googling indicates that when this
occurs it is related to the ability to create a native process in order to interact with the
file system.
I don't think the issue is related to Java heap but rather available RAM on the box. How much
is actually available on the box? You mentioned IOPS - are you running in a virtual cloud
Using remote storage such as Amazon EBS?

> On May 4, 2016, at 8:56 AM, Joe Witt <> wrote:
> Dale,
> Where there is a fetch file there is usually a list file.  And while
> the symptom of memory issues is showing up in fetch file i am curious
> if the issue might actually be caused in ListFile.  How many files are
> in the directory being listed?
> Mark,
> Are we using a stream friendly API to list files and do we know if
> that API on all platforms really doing things in a stream friendly
> way?
> THanks
> Joe
> On Wed, May 4, 2016 at 7:37 AM, dale.chang13 <> wrote:
>> So I still haven't decrypted this problem, and I am assuming that this is an
>> IOPS problem instead of a RAM issue.
>> I have monitored the memory of the nodes in my cluster during the flow,
>> before and after the "cannot allocate memory" exception occurs. However,
>> there is no memory leak because the memory used by the JVM remains steady
>> between 50 and 100 MB used using jconsole. As a note, I have allocated 1 GB
>> as a minimum and 4 GB as a maximum for the heap size for each node.
>> There are also no changes to the number of active threads (35) in jconsole
>> while the NiFi gui shows up to 20 active threads. Additionally the number of
>> classes loaded and CPU usage remains the same throughout the whole NiFi
>> operation.
>> The only difference I have seen is disk activity on the drive that is
>> configured to read to/write from NiFi.
>> My question is: does it make sense that this is an IO issue, or a RAM/memory
>> issue?
>> --
>> View this message in context:
>> Sent from the Apache NiFi Developer List mailing list archive at

View raw message