flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ge Wu <andrew.ge...@eniro.com>
Subject Re: Limit buffer size for a job
Date Wed, 13 Apr 2016 13:57:34 GMT
Thanks guys for the explanation, I will give it a try.
After all buffers are filled, the back pressure did it’s job, it works so far so good, but
I will defiantly give a try to control the latency.


Thanks again!


Andrew



> On 11 Apr 2016, at 18:19, Stephan Ewen <sewen@apache.org> wrote:
> 
> Hi!
> 
> Ufuk's suggestion explains how to buffer less between Flink operators.
> 
> Is that what you were looking for, or are you looking for a way to fetch more fine grained
in the source from the message queue?
> What type of source are you using?
> 
> Greetings,
> Stephan
> 
> 
> 
> 
> On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi <uce@apache.org <mailto:uce@apache.org>>
wrote:
> Hey Andrew,
> 
> take a look at this here:
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency
<https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency>
> 
> Does this help?
> 
> – Ufuk
> 
> On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <andrew.ge-wu@eniro.com <mailto:andrew.ge-wu@eniro.com>>
wrote:
> > Hi guys
> >
> > We have a prioritized queue, where high priority item can jump the queue and we
do not want to cache too much record in the buffer.
> > Is there a way to configure my streaming source to use less buffer? so source always
fetch and get latest high prio records?
> >
> > Any suggestion? thanks!
> >
> >
> > Andrew
> > --
> > Confidentiality Notice: This e-mail transmission may contain confidential
> > or legally privileged information that is intended only for the individual
> > or entity named in the e-mail address. If you are not the intended
> > recipient, you are hereby notified that any disclosure, copying,
> > distribution, or reliance upon the contents of this e-mail is strictly
> > prohibited and may be unlawful. If you have received this e-mail in error,
> > please notify the sender immediately by return e-mail and delete all copies
> > of this message.
> 


-- 
Confidentiality Notice: This e-mail transmission may contain confidential 
or legally privileged information that is intended only for the individual 
or entity named in the e-mail address. If you are not the intended 
recipient, you are hereby notified that any disclosure, copying, 
distribution, or reliance upon the contents of this e-mail is strictly 
prohibited and may be unlawful. If you have received this e-mail in error, 
please notify the sender immediately by return e-mail and delete all copies 
of this message.

Mime
View raw message