nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: nifi memory question
Date Tue, 07 Jun 2016 15:16:57 GMT
Joe,

The JIRA was NIFI-1008 [1]. I looked into this a couple of times but ended up going
down very deep rabbit holes and not getting the desired impact. I think we could probably
use some refactoring of StandardProcessSession and FileSystemSwapManager to clean
up some of the existing code in such a way that this would become easier to tackle.

Thanks
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-1008 <https://issues.apache.org/jira/browse/NIFI-1008>


> On Jun 7, 2016, at 8:37 AM, Joe Witt <joe.witt@gmail.com> wrote:
> 
> Leslie
> 
> You could update the flowfile queue swap threshold higher in such a case.  That said
at that size of a heap you may see undesirable GC behavior. 
> 
> Simply put what you saw when making tons of tiny flowfiles after a split and it causing
slowness is a bug and should be fine on far smaller heaps. We can and should push those things
out of the heap for large sessions like that.  You should be able to use it like you did and
see no perf implications.  That we must fix. 
> 
> Mark Payne...do you recall the jira number to tackle this?
> 
> Thanks
> Joe
> 
> On Jun 7, 2016 6:23 AM, "Leslie Hartman" <lphartm@nsa.gov <mailto:lphartm@nsa.gov>>
wrote:
> Hello:
> 
>     If you have a server with say 64 Gigs of memory and wanted to give
> 32 Gigs of memory to nifi, what options would you change?
> 
>     When I was splitting a large file up into a lot of smaller pieces it ran out of memory.
> 
>     Thank You.
> 
>     Leslie Hartman


Mime
View raw message