nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: back pressure
Date Tue, 31 May 2016 15:52:51 GMT
Hi Pradeep,

Back-pressure tells the source component to no longer execute when the
threshold has been reached.

For example, say you have a GetFile -> PutHDFS and you configure and object
threshold of 100 on the queue between them.
If the queue gets to 100, GetFile will no longer be allowed to run until
the queue drops below 100. As soon as it drops below the threshold it
starts running again and repeating the process.
If you use the data size, it is the total size of all flow files in queue,
so in the previous example it could be set to 100 MB and then GetFile would
stop executing when the queue has 100 MB worth of data.

Overall back-pressure is a way to respond to downstream components that are
taking longer to process, and slow down the source components so they don't
keep producing more data.

Hope this helps.

-Bryan


On Tue, May 31, 2016 at 10:29 AM, pradeepbill <pradeep.bill@gmail.com>
wrote:

> hi there, I am trying to understand what the below text says from NIFI
> documentation,
>
> 1. What does back pressure mean really in NIFI?, here is my simple flow,
> Some NIFI-source -> Some NIFI processor.
> If the amount of input data from the  source is more than the amount of
> data
> the processor can process , then back pressure is applied ? and the 2
> parameters(“Back pressure object threshold.”, “Back pressure data size
> threshold.”) help in that case ?
>
> 2.If back pressure is applied and I assume flow files are queued up, and
> sent in the order they were received.Please confirm
>
> 2.back pressure is applied by default ?
>
>
> "NiFi provides two configuration elements for Back Pressure. These
> thresholds indicate how much data should be allowed to exist in the queue
> before the component that is the source of the Connection is no longer
> scheduled to run. This allows the system to avoid being overrun with data.
> The first option provided is the “Back pressure object threshold.” This is
> the number of FlowFiles that can be in the queue before back pressure is
> applied. The second configuration option is the “Back pressure data size
> threshold.” This specifies the maximum amount of data (in size) that should
> be queued up before applying back pressure. This value is configured by
> entering a number followed by a data size (B for bytes, KB for kilobytes,
> MB
> for megabytes, GB for gigabytes, or TB for terabytes)."
>
>
>
> Thanks
> Pradeep
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/back-pressure-tp10801.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message