flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dan bress <danbr...@gmail.com>
Subject Re: currentLowWatermark metric not reported for all tasks?
Date Fri, 30 Sep 2016 16:13:35 GMT
Awesome!  It would definitely help me troubleshoot lagging watermarks if i
can see what watermark all my sources have seen.  Thanks for looking into
this!

Dan

On Fri, Sep 30, 2016 at 5:48 AM Stephan Ewen <sewen@apache.org> wrote:

> I think what you describe, Chesnay, is exactly what we should do...
>
> On Fri, Sep 30, 2016 at 1:15 PM, Chesnay Schepler <chesnay@apache.org>
> wrote:
>
> > Hello Dan,
> >
> > the technical reason is that this metric is only collected in the
> > *InputProcessor classes, which aren't used for source tasks.
> >
> > I do recall that there were discussions about source watermarks, but
> > frankly i don't remember why we didn't add them.
> >
> > In order to add them one would only have to modify the SourceContext
> > classes to a) store the last emitted watermark and b) expose it through a
> > metric.
> >
> > Regards,
> > Chesnay
> >
> >
> > On 29.09.2016 23:03, dan bress wrote:
> >
> >> I am trying to measure the currentLowWatermark throughout my dataflow,
> but
> >> I am not seeing it for tasks with sources.  For those tasks I see these
> >> metrics:
> >>
> >> lastCheckpointSize
> >> numBytesInLocal
> >> numBytesInRemote
> >> numBytesOut
> >>
> >> why am I not seeing currentLowWatermark on these tasks?
> >>
> >> Thanks!
> >>
> >> Dan
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message