nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran <b.deep.internatio...@gmail.com>
Subject MergeContent across a NiFi cluster
Date Wed, 15 Feb 2017 21:29:07 GMT
Hello,

I need to send data from one organisation to another but there are data
size limits between them (this isn't my choice and has been enforced on
me). I've got a 4 node NiFi cluster in each organisation.

The sending NiFi cluster has the following data flow:
Ingest the data by various means
    -> Compress Data using CompressContent
      -> If file size > X amount I use SplitContent
        -> HTTPS POST to load balancer sitting in front of the NiFi
cluster in the other organisation

On the receiving NiFi cluster I wanted to:
-> Receive the data
    -> MergeContent
      -> Do what ever else with the data...

The problem I can't get round is that if I split the content into 3
fragments and send them to the receiving NiFi instance because it's
behind a load balancer I can't guarantee that the 3 fragments are
received by the same node.

Q1) I'm assuming that for MergeContent to work all the fragments of a
single piece of data have to arrive on the same NiFi node or is there a
option to have it working across a cluster?

Q2) How long does the MergeContent processor wait for all the fragments?
If one of the fragments gets lost does it timeout after a certain
period?

I was thinking one way to solve this of to have the HTTPListener on the
receiving NiFi only listening on the primary node which would ensure all
the fragments arrive on the same node. The downside would be that I end
up with idle NiFi nodes.

Is there anything obvious that I'm missed that would solve my issue?

Thanks in advance,

Brian

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Mime
View raw message