flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ciar, David B." <dcia...@ceh.ac.uk>
Subject Window Functions with Incremental Aggregation
Date Mon, 01 Aug 2016 16:22:35 GMT
Hello everyone,


I'm trying to understand how I can use the Incremental Aggregation + Window Functions, as
I've been unsuccessfully trying for a while now.


The use-case is one where I have a stream of objects, I want to count the number of objects
within a sliding window, and then within the window compare the count against a Redis threshold
value, then emit a new type of object with the count, the window meta-data, and a boolean
of whether the threshold was reached.


The individual parts I understand, but it is getting the fold and window function correct
that is giving me trouble.  The stripped down code for example is here:


https://gist.github.com/dbciar/36e4cf7df6eae2c214efb72d30385fd8


I think one issue is that what I want to do may not be possible, as when I look at the apply(accumulator,
fold, window) definition here:


https://github.com/apache/flink/blob/09e28109b3707f23230ea50526b7db8aa166df15/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedStream.scala


..I think that the apply function needs to output a DataStream of the same type that the fold
function uses as an aggregator, is this correct?


At the moment I get the error that apply does not have a matching signature, even when I changed
the window function to return an Int.


I'm using version 1.1.0 artefacts from https://repository.apache.org/content/repositories/orgapacheflink-1098/.


Any help or links would be great, I've tried searching the mailing list but I wasn't able
to find anything that's close to this situation that I could use,


Thanks

David


________________________________
This message (and any attachments) is for the recipient only. NERC is subject to the Freedom
of Information Act 2000 and the contents of this email and any reply you make may be disclosed
by NERC unless it is exempt from release under the Act. Any material supplied to NERC may
be stored in an electronic records management system.
________________________________

Mime
View raw message