nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: FetchFile Cannot Allocate Enough Memory
Date Wed, 04 May 2016 13:06:25 GMT
ListFile performs a listing using Java's File.listFiles(). This will provide a list of all
files in the
directory. I do not believe this to be related, though. Googling indicates that when this
error
occurs it is related to the ability to create a native process in order to interact with the
file system.
I don't think the issue is related to Java heap but rather available RAM on the box. How much
RAM
is actually available on the box? You mentioned IOPS - are you running in a virtual cloud
environment?
Using remote storage such as Amazon EBS?

> On May 4, 2016, at 8:56 AM, Joe Witt <joe.witt@gmail.com> wrote:
> 
> Dale,
> 
> Where there is a fetch file there is usually a list file.  And while
> the symptom of memory issues is showing up in fetch file i am curious
> if the issue might actually be caused in ListFile.  How many files are
> in the directory being listed?
> 
> Mark,
> 
> Are we using a stream friendly API to list files and do we know if
> that API on all platforms really doing things in a stream friendly
> way?
> 
> THanks
> Joe
> 
> On Wed, May 4, 2016 at 7:37 AM, dale.chang13 <dale.chang13@outlook.com> wrote:
>> So I still haven't decrypted this problem, and I am assuming that this is an
>> IOPS problem instead of a RAM issue.
>> 
>> I have monitored the memory of the nodes in my cluster during the flow,
>> before and after the "cannot allocate memory" exception occurs. However,
>> there is no memory leak because the memory used by the JVM remains steady
>> between 50 and 100 MB used using jconsole. As a note, I have allocated 1 GB
>> as a minimum and 4 GB as a maximum for the heap size for each node.
>> 
>> There are also no changes to the number of active threads (35) in jconsole
>> while the NiFi gui shows up to 20 active threads. Additionally the number of
>> classes loaded and CPU usage remains the same throughout the whole NiFi
>> operation.
>> 
>> The only difference I have seen is disk activity on the drive that is
>> configured to read to/write from NiFi.
>> 
>> My question is: does it make sense that this is an IO issue, or a RAM/memory
>> issue?
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/FetchFile-Cannot-Allocate-Enough-Memory-tp9720p9901.html
>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.


Mime
View raw message