flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Debugging watermarks?
Date Sat, 21 May 2016 15:15:05 GMT
Hi Niels,
you can log the watermarks by implementing a custom operator. (Operators
have access to the watermarks.) The map operator is a good example for this:

public class StreamMap<IN, OUT>
        extends AbstractUdfStreamOperator<OUT, MapFunction<IN, OUT>>
        implements OneInputStreamOperator<IN, OUT> {

    private static final long serialVersionUID = 1L;

    public StreamMap(MapFunction<IN, OUT> mapper) {
        chainingStrategy = ChainingStrategy.ALWAYS;

    public void processElement(StreamRecord<IN> element) throws Exception {


    public void processWatermark(Watermark mark) throws Exception {

In processWatermark() you would print/log the watermark. You can have a
simple identity operator that just forwards and prints and insert it
anywhere in the pipeline.

For your second question this section in the doc might be interesting:


On Sat, 21 May 2016 at 16:05 Niels Basjes <Niels@basjes.nl> wrote:

> Hi,
> I was working on a streaming application last week and I got stuck in a
> situation where I got the same time based window many times.
> I expect that I made a mistake in creating the watermarks in relation to
> the data I have and the watermark generating code.
> Writing the events to the console (for debugging) is easy, yet I have not
> been able to write the watermarks to my console.
> My question is very simple: How do I log the watermarks in the console so
> I can see the data and understand my mistake.
> I would also like to know "where do the watermarks live" in relation to
> the actual data.
> Thanks.
> Niels Basjes

View raw message