Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Subject: Re: Streaming API has a long delay at the beginning of the process.
To: Till Rohrmann <trohrmann@apache.org>, Fabian Hueske <fhueske@gmail.com>
Cc: user <user@flink.apache.org>
References: <CAOmjRb1eHoWHnhLGHmYjQ5gMU=CgNWLgeSRYSmvOzNwh5a1G1Q@mail.gmail.com>
 <8f88c5ed-2e33-ceee-3f4e-b99ca1b2f635@kddi-research.jp>
 <CAAdrtT13FMr1MtRH0RMchf5YkQ1wyTPwVpYuV=1s6VUefsyRaQ@mail.gmail.com>
From: Yuta Morisawa <yu-morisawa@kddi-research.jp>
Message-ID: <5727a1df-6bb3-2705-1093-4fbadc4a97a7@kddi-research.jp>
Date: Fri, 15 Sep 2017 11:25:15 +0900
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.3.0
MIME-Version: 1.0
In-Reply-To: <CAAdrtT13FMr1MtRH0RMchf5YkQ1wyTPwVpYuV=1s6VUefsyRaQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
archived-at: Fri, 15 Sep 2017 02:25:23 -0000

Hi, Fabian

 > If I understand you correctly, the problem is only for the first events
 > that are processed.
Yes. More Precisely, first 300 kafka-messages.

 > AFAIK, Flink lazily instantiates its operators which means that a source
 > task starts to consume records from Kafka before the subsequent tasks
 > have been started.
That's a great indication. It describe well the affair.
But, according to the document, it says "The operations are actually 
executed when the execution is explicitly triggered by an execute() call 
on the execution environment.".
What does it mean?
AFAIK, common Flink programs invoke execute() in main().
Every operators start at this time? I think maybe no.

- Flink Document
 
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/api_concepts.html#lazy-evaluation


 > Not sure if or what can be done about this behavior.
 > I'll loop in Till who knows more about the lifecycle of tasks.
Thank you very much for your kindness.

Regards, Yuta

On 2017/09/14 19:32, Fabian Hueske wrote:
> Hi,
> 
> If I understand you correctly, the problem is only for the first events 
> that are processed.
> 
> AFAIK, Flink lazily instantiates its operators which means that a source 
> task starts to consume records from Kafka before the subsequent tasks 
> have been started.
> That's why the latency of the first records is higher.
> 
> Not sure if or what can be done about this behavior.
> I'll loop in Till who knows more about the lifecycle of tasks.
> 
> Best, Fabian
> 
> 
> 2017-09-12 11:02 GMT+02:00 Yuta Morisawa <yu-morisawa@kddi-research.jp 
> <mailto:yu-morisawa@kddi-research.jp>>:
> 
>     Hi,
> 
>     I am worrying about the delay of the Streaming API.
>     My application is that it gets data from kafka-connectors and
>     process them, then push data to kafka-producers.
>     The problem is that the app suffers a long delay when the first data
>     come in the cluster.
>     It takes about 1000ms to process data (I measure the time with
>     kafka-timestamp). On the other hand, it works well after 2-3 seconds
>     first data come in (the delay is about 200ms).
> 
>     The application is so delay sensitive that I want to solve this problem.
>     Now, I think this is a matter of JVM but I have no idea to
>     investigate it.
>     Is there any way to avoid this delay?
> 
> 
> 
>     Thank you for your attention
>     Yuta
> 
>