flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Baghino <stefano.bagh...@radicalbit.io>
Subject Re: Finding the average temperature
Date Sat, 13 Feb 2016 18:11:58 GMT
Hi Nirmalaya,

I think for your use case in particular it's enough to specify the reducer
that computes the average to have a parallelism of 1 by calling the
`setParallelism` API when you apply it. Keep in mind that you can still
enjoy a high level of parallelism up until the last operator by using a
combiner, which is basically a reducer that operates locally. This way, you
can compute 4 partial averages in parallel and then have the final reducer
actually reporting the average on the whole dataset.

I hope I have been helpful.

On Sat, Feb 13, 2016 at 7:03 PM, Nirmalya Sengupta <
sengupta.nirmalya@gmail.com> wrote:

> Hello Flinksters,
> This is perhaps too trivial for most here in this forum, but I want to
> have my understanding clear.
> I want to find the average of temperatures coming in as a stream. The
> tuple as the following fields:
> probeID,readingTimeStamp,radiationLevel,photoSensor,humidity,ambientTemperature
> The 'readingTimeStamp' field provides the 'Event Time', and this provides
> the basis of the timeWindow(10) that I employ.
> I _keyBy_ the 'readingTimeStamp' and collect readings for next 10 seconds,
> and then I compute the average of the 'ambientTemperature'. After every 10
> seconds, I want to be able to find the average temperature *so far* (a
> single value).
> Because my application defaults to parellelism == 4 (my laptop is 4 core),
> my understanding is that just by using a combination of an appropriate
> RichXXXFunction() and saving State in the RuntimeContext, I may not get the
> correct result. This is because, depending the way the Keys are
> distributed, 4 _different_  averages will be produced. Is this
> understanding right? If so, is using a *Broadcast variable*, the solution?
> Please help me plug the gap in understanding, if any.
> -- Nirmalya
> --
> Software Technologist
> http://www.linkedin.com/in/nirmalyasengupta
> "If you have built castles in the air, your work need not be lost. That is
> where they should be.
> Now put the foundation under them."

Stefano Baghino

Software Engineer @ Radicalbit

View raw message