flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Vararu <vadim.var...@adswizz.com>
Subject Re: Is it OK to have very many session windows?
Date Mon, 20 Feb 2017 15:58:02 GMT
It's something like:

DataStreamSource<String> stream = env.addSource(getKafkaConsumer(parameterTool)); stream
         .map(getEventToDomainMapper())
         .keyBy(getKeySelector())
         .window(ProcessingTimeSessionWindows.withGap(Time.minutes(90)))
         .reduce(getReducer())
         .map(getToJsonMapper())
         .addSink(getKafkaProducer(parameterTool));


Each new event may be reduced against the existent state from the 
window, so normally it's okay to have only 1 row in memory.

I'm new to Flink and have not yet reached the "incremental aggregates" 
but, if i understand correctly, it fits my case.

Vadim.



On 20.02.2017 17:47, Timo Walther wrote:
> Hi Vadim,
>
> this of course depends on your use case. The question is how large is 
> your state per pane and how much memory is available for Flink?
> Are you using incremental aggregates such that only the aggregated 
> value per pane has to be kept in memory?
>
> Regards,
> Timo
>
>
> Am 20/02/17 um 16:34 schrieb Vadim Vararu:
>> HI guys,
>>
>> Is it okay to have very many (tens of thousands or hundreds of 
>> thousand) of session windows?
>>
>>
>> Thanks, Vadim.
>
>


Mime
View raw message