nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toivo Adams <>
Subject How to implement Scatter-Gather?
Date Thu, 18 Jun 2015 09:30:11 GMT
Please see

My use case is little bit different but overall pattern is same.
Instead of sending requests to different vendors some data must be
Processors Calc_A, Calc_B, Calc_C and Calc_D calculate values A, B, C and D.
All processors are independent from each other. Each processor use only data
from request as input.
Every request contains unique requestId which will be attached to every
FlowFile during processing.

How to implement this in NiFi?

1. Sequential processing
Listener  ->   Calc_A   ->    Calc_B   ->    Calc_C   ->   Calc_D   -> 
BuildResponse   ->  SendResponse

Each Calc_* processor will add calculated value to FlowFile.
So After Calc_B  FlowFile contains A and B values.
And After Calc_C  FlowFile contains A, B and C values.
BuildResponse incoming FlowFile contains A, B, C and D values and building
response is easy.

Sequential processing works but when Calc_* processors are slow overall
service latency is too big.

2. Parallel processing
After Listener we can send request FlowFile to all Calc_* processors
Problem is how to aggregate values A, B, C and D from different processors?

This question may belong to user list, but I thought solutions are probably
not trivial and maybe some overall development is reasonable?


View this message in context:
Sent from the Apache NiFi (incubating) Developer List mailing list archive at

View raw message