Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
References: <1499883755759-14229.post@n4.nabble.com>
In-Reply-To: <1499883755759-14229.post@n4.nabble.com>
From: =?UTF-8?Q?Gyula_F=C3=B3ra?= <gyula.fora@gmail.com>
Date: Wed, 12 Jul 2017 21:21:54 +0000
Message-ID: <CA+faj9ww5_7TtOrPz7bjZJJarixDzHo0J+moAqJH2xK9_Sfvaw@mail.gmail.com>
Subject: Re: sanity check in production
To: burgesschen <tchen325@bloomberg.net>, user@flink.apache.org
Content-Type: multipart/alternative; boundary="001a114dd3405a77390554256874"
archived-at: Wed, 12 Jul 2017 21:39:03 -0000

--001a114dd3405a77390554256874
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi!

Assuming you have some spare compute resources on your cluster (which you
should have in a production setting to be safe). I think 2) would be the
best option, ideally started from a savepoint of the production job to
verify your state logic as well.

You could also run the test job on a smaller parallelism setting, and
verify that it actually works, then maybe run some live data through it as
well before killing of the test job and updating the prod job.

Even though this might have a fairly high temporary cost I think it is
ultimately worth it to test on live data before upgrading the production
job.

Cheers,
Gyula

burgesschen <tchen325@bloomberg.net> ezt =C3=ADrta (id=C5=91pont: 2017. j=
=C3=BAl. 12.,
Sze, 20:49):

> Hello everyone,
>
> Our team ran into an issue that testing new deployment of flink job is
> difficult as explained below
>
>
> Goal:
> When we are deploying new version of a flink job in production. we want t=
o
> be able to have the job process some test messages and verify the output =
to
> make sure that the job is running correctly. (sanity check)
>
> Problem:
> The tests messages interfere with the watermark of the flink job,
> potentially causing it dropping real messages.
>
> Possible solutions:
> 1. have a separate watermark for the test messages
>   (looks not supported by the current framework)
>
> 2. run a separate Flink job (same code) in production for sanity check
> before actual deployment
>   (high operational costs)
>
> 3. cancel the running production job with a save point, run a new job wit=
h
> the save point, do sanity check and mess up the watermark of the new job,
> kill the new job, do actual deployment with the same save point.
>   (high operational costs)
>
> Any idea is appreciated, thanks!
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/sanit=
y-check-in-production-tp14229.html
> Sent from the Apache Flink User Mailing List archive. mailing list archiv=
e
> at Nabble.com.
>

--001a114dd3405a77390554256874
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi!<div><br></div><div>Assuming you have some spare comput=
e resources on your cluster (which you should have in a production setting =
to be safe). I think 2) would be the best option, ideally started from a sa=
vepoint of the production job to verify your state logic as well.</div><div=
><br></div><div>You could also run the test job on a smaller parallelism se=
tting, and verify that it actually works, then maybe run some live data thr=
ough it as well before killing of the test job and updating the prod job.</=
div><div><br></div><div>Even though this might have a fairly high temporary=
 cost I think it is ultimately worth it to test on live data before upgradi=
ng the production job.=C2=A0</div><div><br></div><div>Cheers,</div><div>Gyu=
la</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">burgesschen &=
lt;<a href=3D"mailto:tchen325@bloomberg.net">tchen325@bloomberg.net</a>&gt;=
 ezt =C3=ADrta (id=C5=91pont: 2017. j=C3=BAl. 12., Sze, 20:49):<br></div><b=
lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px =
#ccc solid;padding-left:1ex">Hello everyone,<br>
<br>
Our team ran into an issue that testing new deployment of flink job is<br>
difficult as explained below<br>
<br>
<br>
Goal:<br>
When we are deploying new version of a flink job in production. we want to<=
br>
be able to have the job process some test messages and verify the output to=
<br>
make sure that the job is running correctly. (sanity check)<br>
<br>
Problem:<br>
The tests messages interfere with the watermark of the flink job,<br>
potentially causing it dropping real messages.<br>
<br>
Possible solutions:<br>
1. have a separate watermark for the test messages<br>
=C2=A0 (looks not supported by the current framework)<br>
<br>
2. run a separate Flink job (same code) in production for sanity check<br>
before actual deployment<br>
=C2=A0 (high operational costs)<br>
<br>
3. cancel the running production job with a save point, run a new job with<=
br>
the save point, do sanity check and mess up the watermark of the new job,<b=
r>
kill the new job, do actual deployment with the same save point.<br>
=C2=A0 (high operational costs)<br>
<br>
Any idea is appreciated, thanks!<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href=3D"http://apache-flink-user-mailing-l=
ist-archive.2336050.n4.nabble.com/sanity-check-in-production-tp14229.html" =
rel=3D"noreferrer" target=3D"_blank">http://apache-flink-user-mailing-list-=
archive.2336050.n4.nabble.com/sanity-check-in-production-tp14229.html</a><b=
r>
Sent from the Apache Flink User Mailing List archive. mailing list archive =
at Nabble.com.<br>
</blockquote></div>

--001a114dd3405a77390554256874--