flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From milind parikh <milindspar...@gmail.com>
Subject Looking up values in a metadata store for "condensed" events
Date Tue, 10 May 2016 05:31:04 GMT
I am new to flink. I am wondering what the best practice is for looking up
additional values for entities embedded in an event.

For example, an event might have the engine model #. A metadata store has
the typical rpm characteristics of an engine model #; which can be used in
a subsequent processing of the event. I call the process of event
inflation  "event adornment". Currently I use a Map function transformation
to do the lookup in the Map function through instatiating a connection in
the Map function to the Metadata store per map execution. The thing works;
however having 100K connections is not very ideal (assuming 100k eps). Any
alternative ideas; given the implementing class has to be Serializable?

View raw message