flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Finding the average temperature
Date Thu, 18 Feb 2016 09:47:42 GMT
@Nirmalya: Yes, this is right if you temperatures don’t have any other field on which you
could partition them.

@Stefano: Under some circumstances it would be possible to use a a combiner (I’m using the
name as Hadoop MapReduce would use it, here). When the assignment of elements to windows happens
based on the timestamp in the elements and window triggering happens based on watermark it
is possible to combine locally. The reason is that the elements will end up in the same windows
regardless of the time at which the window is processed so it can be done in two steps. Does
that make sense? It’s a very ad-hoc description and I could make up a drawing or something
if that helped. :D


> On 18 Feb 2016, at 10:04, Stefano Baghino <stefano.baghino@radicalbit.io> wrote:
> 
> Thanks, Aljosha, for the explanation. Isn't there a way to apply the concept of the combiner
to a streaming process?
> 
> On Thu, Feb 18, 2016 at 3:56 AM, Nirmalya Sengupta <sengupta.nirmalya@gmail.com>
wrote:
> Hello Aljoscha  <aljoscha@apache.org>
> 
> Thanks very much for clarifying the role of  Pre-Aggregation (rather, Incr-Aggregation,
now that I understand the intention). It helps me to understand. Thanks to Setfano too, for
keeping at the original question of mine.
> 
> My current understanding is that if I have to compute the average of a streaming set
of _temperatures_ then the *best* way to accomplish this, is by employing one node (or thread,
on my laptop), losing speed but gaining deterministic behaviour in the process. I can decide
to capture the average either by grouping the temperatures by count or by time. Because I
am sliding the window anyway, I don't run the risk of accumulation of elements in the window
and buffer overrun.
> 
> Could  you please confirm if my understanding is correct? I feel happy if I 'understand'
the basis of a design well! :-)
> 
> --  Nirmalya
> -- 
> Software Technologist
> http://www.linkedin.com/in/nirmalyasengupta
> "If you have built castles in the air, your work need not be lost. That is where they
should be.
> Now put the foundation under them."
> 
> 
> 
> -- 
> BR,
> Stefano Baghino
> 
> Software Engineer @ Radicalbit


Mime
View raw message