flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Stearn <Tim.Ste...@sas.com>
Subject Appending Windowed Aggregates to Events
Date Fri, 23 Jun 2017 22:59:27 GMT
Hello All,

I'm *very* new to Flink.  I read through the documentation and played with some sample code,
but I'm struggling to get started with my requirements.

We want to use Flink to maintain windowed aggregates as part of a transaction monitoring application.
 These would use sliding window definitions.  An example would be:  "Total amount for CASH
transactions in the last 5 days".   Here's what I need my Flink application to do:

1.      Prepare for transaction processing by reading historical aggregates and building windows

2.      For each new transaction:

a.      Update the windowed aggregate with the new transaction data

b.      Find the window that matches the incoming time stamp and add the aggregate value to
the transaction

c.      Send enhanced transaction (original fields + aggregates from matching window) to downstream
processor via RabbitMQ or Kafka sink

For every transaction coming in, I want one (and only one) output that contains the original
transaction fields plus the aggregates.

I see how to do the code to create the window assigner and the code that incrementally maintains
the aggregates.  I'm not sure how I could join this back to the original transaction record,
appending the aggregate values from the window that matches the transaction date stamp.  This
seems like a join of some kind to me, but I don't know how to implement in in Flink.

I'm hoping someone could reply with some simple code (or even pseudo code) to get me started
on the "join"  part of the above data flow.  Please let me know if I need to clarify.


Tim Stearn

View raw message