flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Richter <s.rich...@data-artisans.com>
Subject Re: Flink takes 40ms ~ 100ms to proceed from one operator to another?
Date Thu, 20 Sep 2018 14:37:23 GMT
Sorry, forgot the link for reference [1], which is https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/datastream_api.html#controlling-latency

> Am 20.09.2018 um 16:36 schrieb Stefan Richter <s.richter@data-artisans.com>:
> Hi,
> you provide not very much information for this question, e.g. what and how exactly your
measure, if this is a local or distributed setting etc. I assume that it is distributed and
that the cause for your observation is the buffer timeout, i.e. the maximum time that Flink
waits until sending a buffer with just one element, which happens to be 100ms by default.
You can decrease this value to some extend, at to cost of potential loss in throughput, but
I think even values around 5-10ms are ok-ish. See [1] for more details. If you want to reduce
latency between chained operators, you can also try to disable object-reuse:
> StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
> env.getConfig().enableObjectReuse();
> Best,
> Stefan
>> Am 20.09.2018 um 16:03 schrieb James Yu <cyu021@gmail.com <mailto:cyu021@gmail.com>>:
>> The previous email seems unable to display embedded images, let me put on the links.
>> Hi,
>> My team and I try to measure total time spent on our flink job and found out that
Flink takes 40ms ~ 100ms to proceed from one operator to another. I wonder how can we reduce
this transition time.
>> Following DAG represents our job:
>> https://drive.google.com/open?id=1wNR8po-SooAfYtCxU3qUDm4-hm16tclV <https://drive.google.com/open?id=1wNR8po-SooAfYtCxU3qUDm4-hm16tclV>

>> and here is the screenshot of our log:
>> https://drive.google.com/open?id=14PTViZMkhagxeNjOb4R8u3BEr7Ym1xBi <https://drive.google.com/open?id=14PTViZMkhagxeNjOb4R8u3BEr7Ym1xBi>

>> at 19:37:04.564, the job is leaving "Source: Custom Source -> Flat Map"
>> at 19:37:04.605, the job is entering "Co-Flat Map"
>> at 19:37:04.605, the job is leaving "Co-Flat Map"
>> at 19:37:04.705, the job is entering "Co-Flat Map -> ...."
>> at 19:37:04.708, the job is leaving "Co-Flat Map -> ..."
>> both "Co-Flat Map" finishes merely instantly, while most of the execution time is
spent on the transition. Any idea?
>> This is a UTF-8 formatted mail
>> -----------------------------------------------
>> James C.-C.Yu
>> +886988713275
>> +8615618429976

View raw message