flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John O <son...@samsung.com>
Subject RE: watermark does not progress
Date Wed, 15 Aug 2018 09:22:21 GMT
I did some more testing.

Below is a pseudo version of by setup.

kafkaconsumer->
assignTimestampsAndWatermarks(BoundedOutOfOrdernessTimestampExtractor)->
process(print1 ctx.timerService().currentWatermark()) ->
keyBy(_.someProp) ->
process(print2 ctx.timerService().currentWatermark()) ->

I am manually sending monotonically increasing (eventtime ) records to kafka topic.

What I see is in print1 I see expected watermark
But print2 is always Long.MIN

It looks like keyBy wipes out the watermark.

Now, if I run the exact same code on a flink cluster, print2 outputs expected watermark.

Jo


From: Fabian Hueske <fhueske@gmail.com>
Sent: Wednesday, August 15, 2018 2:07 AM
To: vino yang <yanghua1127@gmail.com>
Cc: John O <song.o@samsung.com>; user <user@flink.apache.org>
Subject: Re: watermark does not progress

Hi John,

Watermarks cannot make progress if you have stream partitions that do not carry any data.
What kind of source are you using?

Best,
Fabian

2018-08-15 4:25 GMT+02:00 vino yang <yanghua1127@gmail.com<mailto:yanghua1127@gmail.com>>:
Hi Johe,

In local mode, it should also work.
When you debug, you can set a breakpoint in the getCurrentWatermark method to see if you can
enter the method and if the behavior is what you expect.
What is your source? If you post your code, it might be easier to locate.
In addition, for positioning watermark, you can also refer to this email[1].

[1]: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Debugging-watermarks-td7065.html

Thanks, vino.

John O <song.o@samsung.com<mailto:song.o@samsung.com>> 于2018年8月15日周三
上午9:44写道:
I am noticing that watermark does not progress as expected when running locally in IDE. It
just stays at Long.MIN

I am using EventTime processing and have tried both these time extractors.

•         assignAscendingTimestamps ...

•         assignTimestampsAndWatermarks(BoundedOutOfOrdernessTimestampExtractor) ...

Also, configured the environment as so
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)

If I run the job on a flink cluster, I do see the watermark progress.

Is watermarking not supported in local mode?

Thanks
Jo

Mime
View raw message