flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krzysztof Białek <krzysiek.bia...@gmail.com>
Subject Re: Window with recent messages
Date Thu, 22 Feb 2018 18:55:12 GMT
Hi Fabian,

Thank you for your suggestion. In the meantime I rethought this problem and
implemented alternative solution without using windows at all.
I used plain ProcessFunction with
1. Keyed state (by companyId) - to keep ratings per key
2. EventTime timers - to remove outdated ratings from state and emit
recalculated score immediately

This solution gives results in real-time, windows would delay the results
by 1 day I think.

Regards,
Krzysztof


On Thu, Feb 22, 2018 at 9:44 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> Hi Krzysztof,
>
> you could compute the stats in two stages:
>
> 1) compute an daily window. You should use a ReduceFunction or
> AggreagteFunction here if possible to perform the computation eagerly.
> 2) compute a sliding window of 90 days with a 1 day hop (or 90 rows with a
> 1 row hop).
>
> That will crunch down the data in the first window and the second window
> will combine the pre-aggregated results.
>
> Hope this helps,
> Fabian
>
> 2018-02-19 16:36 GMT+01:00 Krzysztof Białek <krzysiek.bialek@gmail.com>:
>
>> Hi,
>>
>> My app is calculating Companies scores from Ratings given by users.
>> Only ratings from last 90 days should be considered.
>>
>> 1. Is it possible to construct window processing ratings from last 90
>> days?
>> I've started with *misusing* countWindow but this solution looks ugly for
>> me.
>>
>> ratingStream
>>   .filter(new OutdatedRatingsFilter(maxRatingAge))
>>   .keyBy(_.companyId)
>>   .countWindow(0L).trigger(new OnEventTrigger).evictor(new OutdatedRatingsEvictor(maxRatingAge))
>>   .process(ratingFunction)
>>
>>
>> 2. How to recalculate score once the rating expires (after 90 days)?
>> I don't want to put artificial ratings into the stream to trigger the
>> recalculation.
>>
>> Any idea how can I do it better?
>>
>> Regards,
>> Krzysztof
>>
>>
>>
>

Mime
View raw message