flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simone Robutti <simone.robu...@radicalbit.io>
Subject Create a cluster inside Flink
Date Thu, 28 Apr 2016 08:48:15 GMT
Hello everyone,

I'm approaching a rather big and complex integration with an existing
software and I would like to hear the opinion of more experienced users on
how to tackle a few issues.

This software builds a cloud with its own logic. What I need is to keep
these nodes as instances inside the TaskManagers and use these instances to
perform operation with dedicated operators. I need to move tabular data
back and forth from and to Flink's Datasets and be able to call methods on
these instances.

I would like to receive suggestions on how to implement this behaviour.
First I thought about using Flink's actor system but I discovered it is not
accessible. So I would like to understand how to properly create these
instances (new thread inside a mapPartition?), how to call methods on them
(create a custom context?) and convert data from a Dataset or a Table to
the custom format of this software (this probably won't be much of a
problem, I will write wrappers or at worst replicate the data).

Any suggestion is welcome.

Thanks,

Simone

Mime
View raw message