Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
Date: Thu, 28 Apr 2016 10:48:15 +0200
Message-ID: 
 <CACDjLVg0iKPR_8ckDeLZh3L_zz7yW3iy9FFAqxNeuSBtsHzH+Q@mail.gmail.com>
Subject: Create a cluster inside Flink
From: Simone Robutti <simone.robutti@radicalbit.io>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1140820e4ed87905318796e3

--001a1140820e4ed87905318796e3
Content-Type: text/plain; charset=UTF-8

Hello everyone,

I'm approaching a rather big and complex integration with an existing
software and I would like to hear the opinion of more experienced users on
how to tackle a few issues.

This software builds a cloud with its own logic. What I need is to keep
these nodes as instances inside the TaskManagers and use these instances to
perform operation with dedicated operators. I need to move tabular data
back and forth from and to Flink's Datasets and be able to call methods on
these instances.

I would like to receive suggestions on how to implement this behaviour.
First I thought about using Flink's actor system but I discovered it is not
accessible. So I would like to understand how to properly create these
instances (new thread inside a mapPartition?), how to call methods on them
(create a custom context?) and convert data from a Dataset or a Table to
the custom format of this software (this probably won't be much of a
problem, I will write wrappers or at worst replicate the data).

Any suggestion is welcome.

Thanks,

Simone

--001a1140820e4ed87905318796e3
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello everyone,<div><br></div><div>I&#39;m approaching a r=
ather big and complex integration with an existing software and I would lik=
e to hear the opinion of more experienced users on how to tackle a few issu=
es.</div><div><br></div><div>This software builds a cloud with its own logi=
c. What I need is to keep these nodes as instances inside the TaskManagers =
and use these instances to perform operation with dedicated operators. I ne=
ed to move tabular data back and forth from and to Flink&#39;s Datasets and=
 be able to call methods on these instances. =C2=A0</div><div><br></div><di=
v>I would like to receive suggestions on how to implement this behaviour. F=
irst I thought about using Flink&#39;s actor system but I discovered it is =
not accessible. So I would like to understand how to properly create these =
instances (new thread inside a mapPartition?), how to call methods on them =
(create a custom context?) and convert data from a Dataset or a Table to th=
e custom format of this software (this probably won&#39;t be much of a prob=
lem, I will write wrappers or at worst replicate the data).</div><div><br><=
/div><div>Any suggestion is welcome.</div><div><br></div><div>Thanks,</div>=
<div><br></div><div>Simone</div></div>

--001a1140820e4ed87905318796e3--