flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Hoang <a...@parcelperform.com>
Subject Re: Time extracting in flink
Date Mon, 22 Jul 2019 10:41:13 GMT
Thanks Biao, just want to not reinvent the wheel :)

> On Jul 22, 2019, at 4:29 PM, Biao Liu <mmyy1110@gmail.com> wrote:
> Hi Andy,
> As far as I know, Flink does not support feature like that. 
> I would suggest recording and calculating the time in user code.
> For example, add a timestamp field (maybe an array) in your record with printing a timestamp
on in by each processing.
> Andy Hoang <andy@parcelperform.com <mailto:andy@parcelperform.com>> 于2019年7月22日周一
> Hi guys,
> I’m trying to write elk log for flink, this help us to store/calculate processing time
of a group of operators for business auditing.
> I read about process_function and Debugging Windows & Event Time in docs. They’re
focus on “keyed” events and monitoring using web/metric, where I want to, for example:
> val stream = env.add_source(kafka_source).map(doX).map(doY).filter(Z)
> Async.unordedWait(stream, calApiBlah).add_source(kafka_source)
> Track total time from `doY` till before add_source operator(). This kind of time can
be processing time,
> or I want to track total time from source to sink of a stream event  (maybe disgesting
> So the target is to retrieve out total time of a set of chosen operators
> I think flink should have supported it somehow, but its just me haven’t found it in
the docs yet.
> Thanks,
> Andy,

View raw message