Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAMbC8iXLFbgAV=uW=x1u7jymiYTH=b51GrWqgY2nzNFcQ3-NBQ@mail.gmail.com>
References: <CAMbC8iXLFbgAV=uW=x1u7jymiYTH=b51GrWqgY2nzNFcQ3-NBQ@mail.gmail.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Wed, 1 Nov 2017 08:57:11 +0100
Message-ID: <CAAdrtT03K1ewD=7ZE5FvTSWtsniQCLmukdRLqGsZJZKzNB6aiQ@mail.gmail.com>
Subject: Re: Batch job per stream message?
To: Tomas Mazukna <tomas.mazukna@gmail.com>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="001a114dba6c803efb055ce73a5d"
archived-at: Wed, 01 Nov 2017 07:57:58 -0000

--001a114dba6c803efb055ce73a5d
Content-Type: text/plain; charset="UTF-8"

Hi Tomas,

triggering a batch DataSet job from a DataStream program for each input
record doesn't sound like a good idea to me.
You would have to make sure that the cluster always has sufficient
resources and handle failures.

It would be preferable to have all data processing in a DataStream job. You
mentioned that the challenge is to join the data of the files with a JDBC
database.
I see two ways to do that in a DataStream program:
- replicate the JDBC table in a stateful operator. This means that you have
to publish updates on the database to the Flink program.
- query the JDBC table with an AsyncFunction. This operator concurrently
executes multiple calls to an external service which improves latency and
throughput. The operator ensures that checkpoints and watermarks are
correctly handled.

Best, Fabian

2017-10-30 19:11 GMT+01:00 Tomas Mazukna <tomas.mazukna@gmail.com>:

> Trying to figure out the best design in Flink.
> Reading from a kafka topic which has messages with pointers to files to be
> processed.
> I am thinking to somehow kick off a batch job per file... unless there is
> an easy way to get a separate dataset per file.
> I can do almost all of this in the stream, parse file with flat map ->
> explode its contents into multiple data elements -> map, etc...
> On of these steps would be to grab another dataset from JDBC source and
> join with the stream's contents...
> I think I am mixing the two concepts here and the right approach would be
> to kick of this mini batch job per file,
> where I have file datase t+ jdbc dataset to join with.
>
> So how would I go about kicking a batch from from streaming job?
>
> Thanks,
> Tomas
>

--001a114dba6c803efb055ce73a5d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi Tomas,<br><br></div>triggering a batch DataSe=
t job from a DataStream program for each input record doesn&#39;t sound lik=
e a good idea to me. <br></div><div>You would have to make sure that the cl=
uster always has sufficient resources and handle failures.</div><div><br></=
div><div>It would be preferable to have all data processing in a DataStream=
 job. You mentioned that the challenge is to join the data of the files wit=
h a JDBC database.</div><div>I see two ways to do that in a DataStream prog=
ram:</div><div>- replicate the JDBC table in a stateful operator. This mean=
s that you have to publish updates on the database to the Flink program.</d=
iv><div>- query the JDBC table with an AsyncFunction. This operator concurr=
ently executes multiple calls to an external service which improves latency=
 and throughput. The operator ensures that checkpoints and watermarks are c=
orrectly handled.</div><div><br></div><div>Best, Fabian<br></div></div><div=
 class=3D"gmail_extra"><br><div class=3D"gmail_quote">2017-10-30 19:11 GMT+=
01:00 Tomas Mazukna <span dir=3D"ltr">&lt;<a href=3D"mailto:tomas.mazukna@g=
mail.com" target=3D"_blank">tomas.mazukna@gmail.com</a>&gt;</span>:<br><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #c=
cc solid;padding-left:1ex"><div dir=3D"ltr">Trying to figure out the best d=
esign in Flink.<div>Reading from a kafka topic which has messages with poin=
ters to files to be processed.</div><div>I am thinking to somehow kick off =
a batch job per file... unless there is an easy way to get a separate datas=
et per file.</div><div>I can do almost all of this in the stream, parse fil=
e with flat map -&gt; explode its contents into multiple data elements -&gt=
; map, etc...</div><div>On of these steps would be to grab another dataset =
from JDBC source and join with the stream&#39;s contents...</div><div>I thi=
nk I am mixing the two concepts here and the right approach would be to kic=
k of this mini batch job per file,=C2=A0</div><div>where I have file datase=
 t+ jdbc dataset to join with.</div><div><br></div><div>So how would I go a=
bout kicking a batch from from streaming job?</div><div><br></div><div>Than=
ks,<br>Tomas</div></div>
</blockquote></div><br></div>

--001a114dba6c803efb055ce73a5d--