flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Proper way to establish bucket counts
Date Fri, 04 Aug 2017 20:16:12 GMT
Hi Robert,
That's right.

The count's are on a per operator-level. I think you can get down to the
task-level but counts per bucket are not tracked.

Maybe Chesnay (in CC) can help here. He knows the metrics system the best.
@Chesnay, is there a way to expire metric counters?
Alternatively, you could think about forking of a stateful map (or
ProcessFunction) in front of the sink that evaluates the bucketing logic
and keeps track of the counts. Then you'd have the bucket counts as another

2017-08-03 2:49 GMT+02:00 Robert Rapplean <rrapplean@altitudedigital.com>:

> That looks like something that could be used to count the total events in
> the overall sink, but doesn't look like it works on a per-bucket basis.
> We're going to try adding a counter to the bucket status object.
> On Wed, Aug 2, 2017 at 3:02 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>> Hi Robert,
>> Flink collects many metrics by default, including the number of records /
>> events that go into each operator (see [1], System Metrics, IO,
>> "numRecordsIn").
>> So, you would only need to access that metric.
>> Best, Fabian
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>> monitoring/metrics.html
>> 2017-08-01 18:56 GMT+02:00 Robert Rapplean <rrapplean@altitudedigital.com
>> >:
>>> I want a count of events that are put into a bucketing sink, but can't
>>> find a ready-made way of doing that. Is there an easier way than to
>>> implement a counter for each bucket via the metrics? If metrics counters is
>>> the easy way, what do I do to make sure that I don't have a memory leak
>>> from expired counters?
>>> Thanks,
>>> Robert

View raw message