flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Salnikov <quix...@gmail.com>
Subject Implement bunch of transformations applied to same source stream in Apache Flink in parallel and combine result
Date Wed, 11 Oct 2017 21:02:45 GMT

Could you please help me - I'm trying to use Apache Flink for machine
learning tasks with external ensemble/tree libs like XGBoost, so my
workflow will be like this:

   - receive single stream of data which atomic event looks like a simple
   vector event=(X1, X2, X3...Xn) and it can be imagined as POJO fields so
   initially we have DataStream<event> source=...
   - a lot of feature extractions code applied to the same event
source: feature1
   = source.map(X1...Xn) feature2 = source.map(X1...Xn) etc. For simplicity
   lets DataStream<int> feature(i) = source.map() for all features
   - then I need to create a vector with extracted features (feature1,
   feature2, ...featureK) for now it will be 40-50 features, but I'm sure
   it will contain more items in future and easily can contains 100-500
   features and more
   - put these extracted features to dataset/table columns by 10 minutes
   window and run final machine learning task on such 10 minutes data

In simple words I need to apply several quite different map operations to
the same single event in stream and then combine result from all map
functions in single vector.

So for now I can't figure out how to implement final reduce step and run
all feature extraction mapjobs in parallel if possible. I spend several
days on flink docs site, youtube videos, googling, reading Flink's sources
but it seems I'm really stuck here.

The easy solution here will be to use single map operation and run each
feature extraction code sequentially one by one in huge map body, and then
return final vector (Feature1...FeatureK) for each input event. But it
should be crazy and non optimal.

Another solution for each two pair of features use join since all feature
DataStreams has same initial event and same key and only apply some
transformation code, but it looks ugly: write 50 joins code with some window.
And I think that joins and cogroups developed for joining different streams
from different sources and not for such map/reduce operations.

As for me for all map operations here should be a something simple which
I'm missing.

Could you please point me how you guys implement such tasks in Flink, and
if possible with example of code?
PS: I posted this question
PPS: If I will use feature1.union(feature2...featureK) I still need somehow
separate and combine features vector before sink, and preserve order of
final vectors.


View raw message