Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <1471866612870-8596.post@n4.nabble.com>
References: <1471866612870-8596.post@n4.nabble.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Mon, 22 Aug 2016 14:23:10 +0200
Message-ID: <CAAdrtT3PFiSZCGSrK7YHSY_ebAFamXoj7X2=7GfVH3fktbjLUQ@mail.gmail.com>
Subject: Re: Complex batch workflow needs (too) much time to create executionPlan
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0d75144fddbc053aa81ed2
archived-at: Mon, 22 Aug 2016 12:23:47 -0000

--94eb2c0d75144fddbc053aa81ed2
Content-Type: text/plain; charset=UTF-8

Hi Markus,

you might be right, that a lot of time is spend in optimization.
The optimizer enumerates all alternatives and chooses the plan with the
least estimated cost. The degrees of freedom of the optimizer are rather
restricted (execution strategies and the used partitioning & sorting keys.
The order of operators is not optimized). However, your plan contains of
more than 250 operators which is pretty large (in fact, I haven't seen a
Flink plan of this size yet).
I assume that this is only one part of the program that exceeded the 20
minutes of optimization, correct?

In order to verify that Flink is stuck in the optimization phase, you could
take a stacktrace to see which methods the Java process currently executes.

One way to improve the optimization time is to set a few JoinHints to
reduce the degrees of freedom and number of enumerated plans.

Hope this helps,
Fabian

2016-08-22 13:50 GMT+02:00 Markus Nentwig <nentwig@informatik.uni-leipzig.de
>:

> Hello Flink community,
>
> I created a slightly long batch workflow for my use case of clustering
> vertices using
> Flink and Gelly. Executing each of the workflow parts individually (and
> write
> intermediate results to disk) works as suspected.
>
> When combining workflow parts to longer jobs, I noticed that the time
> between
> 'Job Name' time and the actual 'Start Time' in the Flink Dashboard differ.
> With
> longer workflow chains the time difference gets bigger and bigger.  At this
> point, I think that Flink is creating the execution plan which is executed
> directly afterwards. As an example (90% of the workflow combined), I 'wait'
> for
> the execution plan for 77-78 seconds, then the job is accepted for
> execution
> and
> needs another 7-9 seconds to process a small test dataset (<8k vertices
> with
> property values and edges) - each run repeated 3 times. If running only
> env.getExecutionPlan() I will wait similar time for the execution plan. I
> added the
> JSON execution plan to this post. For bigger datasets the execution plan
> creation time and the job execution time grows as well in my scenario.
>
> When I now add a vertex centric iteration to my workflow and start the
> Flink
> job, I don't get a result at all: I stopped the job
> (print execution plan to log) at the following point:
>
> - waited > 20 hours after 'flink run ...'
> - two cores on my machine are at 100% all the time working on the flink job
> - no entry in Flink dashboard at all
> - no entry in log file after these lines:
>
> ###
> [...]
> org.apache.flink.client.CliFrontend            - Starting execution of
> program
> org.apache.flink.client.program.Client         - Starting program in
> interactive mode
> org.apache.flink.api.java.ExecutionEnvironment - The job has 2 registered
> types and 0 default Kryo serializers
> org.apache.flink.optimizer.Optimizer           - The parallelism of nested
> dataflows (such as step functions in iterations) is currently fixed to the
> parallelism of the surrounding operator (the iteration).
> ###
>
> Most likely the workflow could be optimized in many ways to need less time
> at
> certain points (yes, I am not a Flink expert in many places), but I think
> that
> long/complex workflows would still suffer of problems like this.
> Due to the fact that every single step is producing output (and some
> combined
> parts of the workflow do so, too), I currently suspect the Flink optimizer
> /
> execution plan creation to be the problem and therefore ask anyone here if
> you
> have experience with similar behavior. Any suggestions how I could
> successfully
> run long/complex workflows not running in such problems? ;)
>
> If there is not (an instant) 'solution' to the problem I would be still
> interested in opinions and ideas, thanks in advance!
>
> Best,
> Markus
>
> complexExecPlan.json
> <http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/file/n8596/complexExecPlan.json>
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Complex-batch-
> workflow-needs-too-much-time-to-create-executionPlan-tp8596.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

--94eb2c0d75144fddbc053aa81ed2
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi Markus,<br><br></div>you might be right, that=
 a lot of time is spend in optimization. <br>The optimizer enumerates all a=
lternatives and chooses the plan with the least estimated cost. The degrees=
 of freedom of the optimizer are rather restricted (execution strategies an=
d the used partitioning &amp; sorting keys. The order of operators is not o=
ptimized). However, your plan contains of more than 250 operators which is =
pretty large (in fact, I haven&#39;t seen a Flink plan of this size yet).<b=
r></div><div>I assume that this is only one part of the program that exceed=
ed the 20 minutes of optimization, correct?<br><br></div><div>In order to v=
erify that Flink is stuck in the optimization phase, you could take a stack=
trace to see which methods the Java process currently executes.<br><br></di=
v><div>One way to improve the optimization time is to set a few JoinHints t=
o reduce the degrees of freedom and number of enumerated plans.<br></div><d=
iv><br></div><div>Hope this helps,<br></div><div>Fabian<br></div></div><div=
 class=3D"gmail_extra"><br><div class=3D"gmail_quote">2016-08-22 13:50 GMT+=
02:00 Markus Nentwig <span dir=3D"ltr">&lt;<a href=3D"mailto:nentwig@inform=
atik.uni-leipzig.de" target=3D"_blank">nentwig@informatik.uni-leipzig.de</a=
>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex">Hello Flink community,<br>
<br>
I created a slightly long batch workflow for my use case of clustering<br>
vertices using<br>
Flink and Gelly. Executing each of the workflow parts individually (and<br>
write<br>
intermediate results to disk) works as suspected.<br>
<br>
When combining workflow parts to longer jobs, I noticed that the time<br>
between<br>
&#39;Job Name&#39; time and the actual &#39;Start Time&#39; in the Flink Da=
shboard differ.<br>
With<br>
longer workflow chains the time difference gets bigger and bigger.=C2=A0 At=
 this<br>
point, I think that Flink is creating the execution plan which is executed<=
br>
directly afterwards. As an example (90% of the workflow combined), I &#39;w=
ait&#39;<br>
for<br>
the execution plan for 77-78 seconds, then the job is accepted for executio=
n<br>
and<br>
needs another 7-9 seconds to process a small test dataset (&lt;8k vertices =
with<br>
property values and edges) - each run repeated 3 times. If running only<br>
env.getExecutionPlan() I will wait similar time for the execution plan. I<b=
r>
added the<br>
JSON execution plan to this post. For bigger datasets the execution plan<br=
>
creation time and the job execution time grows as well in my scenario.<br>
<br>
When I now add a vertex centric iteration to my workflow and start the Flin=
k<br>
job, I don&#39;t get a result at all: I stopped the job<br>
(print execution plan to log) at the following point:<br>
<br>
- waited &gt; 20 hours after &#39;flink run ...&#39;<br>
- two cores on my machine are at 100% all the time working on the flink job=
<br>
- no entry in Flink dashboard at all<br>
- no entry in log file after these lines:<br>
<br>
###<br>
[...]<br>
org.apache.flink.client.<wbr>CliFrontend=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 - Starting execution of<br>
program<br>
org.apache.flink.client.<wbr>program.Client=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0- Starting program in<br>
interactive mode<br>
org.apache.flink.api.java.<wbr>ExecutionEnvironment - The job has 2 registe=
red<br>
types and 0 default Kryo serializers<br>
org.apache.flink.optimizer.<wbr>Optimizer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0- The parallelism of nested<br>
dataflows (such as step functions in iterations) is currently fixed to the<=
br>
parallelism of the surrounding operator (the iteration).<br>
###<br>
<br>
Most likely the workflow could be optimized in many ways to need less time<=
br>
at<br>
certain points (yes, I am not a Flink expert in many places), but I think<b=
r>
that<br>
long/complex workflows would still suffer of problems like this.<br>
Due to the fact that every single step is producing output (and some<br>
combined<br>
parts of the workflow do so, too), I currently suspect the Flink optimizer =
/<br>
execution plan creation to be the problem and therefore ask anyone here if<=
br>
you<br>
have experience with similar behavior. Any suggestions how I could<br>
successfully<br>
run long/complex workflows not running in such problems? ;)<br>
<br>
If there is not (an instant) &#39;solution&#39; to the problem I would be s=
till<br>
interested in opinions and ideas, thanks in advance!<br>
<br>
Best,<br>
Markus<br>
<br>
complexExecPlan.json<br>
&lt;<a href=3D"http://apache-flink-user-mailing-list-archive.2336050.n4.nab=
ble.com/file/n8596/complexExecPlan.json" rel=3D"noreferrer" target=3D"_blan=
k">http://apache-flink-user-<wbr>mailing-list-archive.2336050.<wbr>n4.nabbl=
e.com/file/n8596/<wbr>complexExecPlan.json</a>&gt;<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href=3D"http://apache-flink-user-mailing-l=
ist-archive.2336050.n4.nabble.com/Complex-batch-workflow-needs-too-much-tim=
e-to-create-executionPlan-tp8596.html" rel=3D"noreferrer" target=3D"_blank"=
>http://apache-flink-user-<wbr>mailing-list-archive.2336050.<wbr>n4.nabble.=
com/Complex-batch-<wbr>workflow-needs-too-much-time-<wbr>to-create-executio=
nPlan-<wbr>tp8596.html</a><br>
Sent from the Apache Flink User Mailing List archive. mailing list archive =
at Nabble.com.<br>
</blockquote></div><br></div>

--94eb2c0d75144fddbc053aa81ed2--