nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ashwin.konale@gmail.com <ashwin.kon...@gmail.com>
Subject Re: Slow flowfile transfer from process group port to output port.
Date Wed, 26 Sep 2018 08:02:58 GMT


On 2018/09/25 15:34:03, Mark Payne <markap14@hotmail.com> wrote: 
> Ashwin,
> 
> You'll want to Right-Click on the RPG and then choose to configure Remote Ports. From
there, you can
> configure how many threads should be used to pull data from each port. So you've updated
the Output Port
> to use up to 3 threads per node to provide the data, but each node is still only using
1 thread to pull the data.
> By configuring the Remote Ports, you can configure the RPG to use 3 threads to pull the
data as well.
> 
> Thanks
> -Mark
> 
> 
> > On Sep 25, 2018, at 11:29 AM, ashwin konale <ashwin.konale@gmail.com> wrote:
> > 
> > Hi I have the following flow which I am trying to increase the throughput.
> > I am runnign 3 node nifi cluster(v1.6)
> > 
> > CDC -> processGroupOutputPort -> externalOutputPort
> > RPG -> downstream
> > 
> > Flowfiles are always queued up between processGroupOutputPort to
> > externalOutputPort. I dont see any configuration to increase the number of
> > threads for this operation as well. I have set backpressure to be 10,000
> > flowfiles. Since the CDC produces at much faster rate than this, I want to
> > increase the throughput of the flow. Also I see in the UI
> > externalOutputPort has number of threads running as 3 all the time no
> > matter how much I increase number of threads to consume in RPG.
> > 
> > Any pointers here will be much helpful.
> > 
> > Ashwin
> 
> Hello Mark, Thanks a lot for the reply.
I have already configured number of threads to in remote ports in RPG. But no matter how many
threads I set it to, the number of threads running in output port is showing as 3. And there
is no setting to update the number of threads for the output port. I quickly went through
LocalPort.java . (https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java)
Looks like it transfers in the batches of 100. Is there any way to increase the throughput.?


Mime
View raw message