flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Flink UI and records sent
Date Wed, 05 Apr 2017 11:52:10 GMT
My job is a batch one.
Here's an image of two different execution of the job.
The third line is where the first(1M) is called. In the left side the count
is what I expect, in the second is slightly less :(

[image: Inline image 1]

Any idea?

On Wed, Apr 5, 2017 at 12:51 PM, Chesnay Schepler <chesnay@apache.org>
wrote:

> Hey Flavio,
>
> it's unlikely that the counters skip a record.
>
> For the webUI these metrics are transported in 2 different ways:
> For running tasks they are fetched through the metric system; this
> provides no guarantee that the final count is ever displayed.
> For finished tasks the final count is stored in the ExecutionGraph and
> should show an accurate final count.
>
> So, the question is in which state your task is.
>
> Regards,
> Chesnay
>
>
> On 05.04.2017 09:55, Flavio Pompermaier wrote:
>
> Hi to all,
> I'm using Flink 1.2.0 and I have a job that (at some point) calls
> dataset.first(1M).
> Sometimes the records sent displayed by the UI are less than 1M (lik
> 999709).
> Is it possible that the UI (or the internal Flink counters) miss some
> record?
>
> Best,
> Flavio
>
> --
> Flavio Pompermaier
> Development Department
>
> OKKAM S.r.l.
> Tel. +(39) 0461 1823908 <+39%200461%20182%203908>
>
>
>


-- 
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 1823908

Mime
View raw message