flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Streaming API has a long delay at the beginning of the process.
Date Thu, 14 Sep 2017 10:32:20 GMT

If I understand you correctly, the problem is only for the first events
that are processed.

AFAIK, Flink lazily instantiates its operators which means that a source
task starts to consume records from Kafka before the subsequent tasks have
been started.
That's why the latency of the first records is higher.

Not sure if or what can be done about this behavior.
I'll loop in Till who knows more about the lifecycle of tasks.

Best, Fabian

2017-09-12 11:02 GMT+02:00 Yuta Morisawa <yu-morisawa@kddi-research.jp>:

> Hi,
> I am worrying about the delay of the Streaming API.
> My application is that it gets data from kafka-connectors and process
> them, then push data to kafka-producers.
> The problem is that the app suffers a long delay when the first data come
> in the cluster.
> It takes about 1000ms to process data (I measure the time with
> kafka-timestamp). On the other hand, it works well after 2-3 seconds first
> data come in (the delay is about 200ms).
> The application is so delay sensitive that I want to solve this problem.
> Now, I think this is a matter of JVM but I have no idea to investigate it.
> Is there any way to avoid this delay?
> Thank you for your attention
> Yuta

View raw message