nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oscar dela Pena <>
Subject NiFi 0.4.1 Very slow processing of flow files using PutKafka
Date Mon, 04 Apr 2016 08:21:00 GMT
Hi NiFi team, 

This is our current NiFi flow: 
Our Kafka is version 0.8.2 and NiFi is version 0.4.0. The two versions should be a match according
to source codes and here. 
However, PutKafka is extremely slow when processing queued flow files, coming at 40GB/hour
We had to add a dynamic property block.on.buffer.full = true to get rid of "BufferExhaustedException",
and set the buffer size to 4GB. 

Flow files are plain text files and are delimited by \n(new line). 
File delimiter is set to \n. 
Everything else are default processor values. 

Previous version of Kafka and NiFi doesn't have this problem. We have another running Kafka
instance to prove it. 
We used PutKafka v0.3.0. We built it from source and renamed the processor. It sends to Kafka
0.8.1 the same messages as PutKafka v0.4.0. 
Same delimiter is used. All messages are sent and no problems. More details on the configurations
are posted below. 

	PutKafka 0.3.0 	PutKafka 0.4.0 
Kafka Version 	0.8.1 	

Errors 	No error. All messages are sent to Kafka. 	
Without block.on.buffer.full = true , BufferExhaustedException occurs. 
I added block.on.buffer.full = as a dynamic property and BufferExhaustedException is gone.

Sending records is really slow. 
120GB worth of data get stuck in queue when running the flow for 2hrs. 


How should we configure our PutKafka so that it can have the same performance as the old PutKafka(0.3.0)


View raw message