flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tanguy Racinet <tanraci...@gmail.com>
Subject Flink Stream assign unique transaction ID
Date Fri, 19 Feb 2016 14:38:43 GMT
Hi,

I am currently working on Flink streaming API and I am trying to assign unique IDs to transactions
to track changes and mine Frequent Item Sets.

I identified several potential solutions such as:
 - Check pointing Local variables (snapshotState and restoreState for once only passage)
 - Key Value State Interface (using operatorState and getRuntimeContext but is deprecated)
 - broadcast function
 - Co-Map function 

After trying all of them, the only one I could actually make work for my needs uses the deprecated
operatorState.
I tried using co-map function but couldn’t the number from one map and use it in the other
mapper to modify the Tid value.

Is there any solution I haven’t thought of in order to assign a unique Long to every one
of my transactions within the DataStream and to ensure they are not only unique but also ordered
and linearly distributed (transaction1 -> id = 1; transaction2 -> id = 2; etc)
Or maybe some working examples of the solutions I tried but couldn’t obtain what I needed
with it ?

Thank you in advance.
Regards,
Tanguy R.
Mime
View raw message