Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAGco--bsH1E_zAKtdQe+rKir0bRWKMRU=5n=uS5z+qvcrQfncA@mail.gmail.com>
References: <48A214C8-C8F9-443A-9D60-6B06CB477086@mail.polimi.it>
	<CAGco--bsH1E_zAKtdQe+rKir0bRWKMRU=5n=uS5z+qvcrQfncA@mail.gmail.com>
Date: Tue, 30 Jun 2015 10:41:21 +0200
Message-ID: 
 <CAAdrtT22NCULcFvsCFvLdJ1WjwjM9O6YsU1tLC24pc6abqVAiw@mail.gmail.com>
Subject: Re: Execution graph
From: Fabian Hueske <fhueske@gmail.com>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11c3d55cbb83bf0519b82b0b

--001a11c3d55cbb83bf0519b82b0b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

As an addition, some operators can only be run with a parallelism of 1. For
example data sources based on collections and (un-grouped) all reduces. In
some cases, the parallelism of the following operators will as well be set
to 1 to avoid a network shuffle.

If you do:

env.fromCollection(myCollection).map(new MyMapper()).groupBy(0).reduce(new
MyReduce()).writeToFile();

the data source and mapper will be run with a parallelism of 1, the reducer
and sink will be executed with the default parallelism.

Best, Fabian

2015-06-30 10:25 GMT+02:00 Maximilian Michels <mxm@apache.org>:

> Hi Michele,
>
> If you don't set the parallelism, the default parallelism is used. For th=
e
> visualization in the web client, a parallelism of one is used. When you r=
un
> your example from your IDE, the default parallelism is set to the number =
of
> (virtual) cores of your CPU.
>
> Moreover, Flink will currently not automatically set the parallelism in a
> cluster environment. It will use the default parallelism or the user-set
> parallelism. In your example, if you set the parallelism explicitly then =
it
> will also show up in the visualization.
>
> Best,
> Max
>
> On Tue, Jun 30, 2015 at 7:11 AM, Michele Bertoni <
> michele1.bertoni@mail.polimi.it> wrote:
>
>> Hi, I was trying to run my program in the flink web environment (the
>> local one)
>> when I run it I get the graph of the planned execution but in each node
>> there is a "parallelism =3D 1=E2=80=9D, instead i think it runs with par=
 =3D 8 (8 core,
>> i  always get 8 output)
>>
>> what does that mean?
>> is that wrong or is it really running with 1 degree of par?
>>
>> just a note: I never do any setParallelism() command, i leave it
>> automatical
>>
>> thanks
>> Best
>> Michele
>
>
>

--001a11c3d55cbb83bf0519b82b0b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>As an addition, some operators can only be run with a=
 parallelism of 1. For example data sources based on collections and (un-gr=
ouped) all reduces. In some cases, the parallelism of the following operato=
rs will as well be set to 1 to avoid a network shuffle.<br><br></div><div>I=
f you do:<br><br></div><div>env.fromCollection(myCollection).map(new MyMapp=
er()).groupBy(0).reduce(new MyReduce()).writeToFile();<br><br></div><div>th=
e data source and mapper will be run with a parallelism of 1, the reducer a=
nd sink will be executed with the default parallelism.<br><br></div><div>Be=
st, Fabian<br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmai=
l_quote">2015-06-30 10:25 GMT+02:00 Maximilian Michels <span dir=3D"ltr">&l=
t;<a href=3D"mailto:mxm@apache.org" target=3D"_blank">mxm@apache.org</a>&gt=
;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;b=
order-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div=
>Hi Michele,<br><br></div>If you don&#39;t set the parallelism, the default=
 parallelism is used. For the visualization in the web client, a parallelis=
m of one is used. When you run your example from your IDE, the default para=
llelism is set to the number of (virtual) cores of your CPU.<br><br></div><=
div>Moreover, Flink will currently not automatically set the parallelism in=
 a cluster environment. It will use the default parallelism or the user-set=
 parallelism. In your example, if you set the parallelism explicitly then i=
t will also show up in the visualization.<br></div><div><br></div>Best,<br>=
</div>Max<br></div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gm=
ail_extra"><br><div class=3D"gmail_quote">On Tue, Jun 30, 2015 at 7:11 AM, =
Michele Bertoni <span dir=3D"ltr">&lt;<a href=3D"mailto:michele1.bertoni@ma=
il.polimi.it" target=3D"_blank">michele1.bertoni@mail.polimi.it</a>&gt;</sp=
an> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left:1px #ccc solid;padding-left:1ex">Hi, I was trying to run my pro=
gram in the flink web environment (the local one)<br>
when I run it I get the graph of the planned execution but in each node the=
re is a &quot;parallelism =3D 1=E2=80=9D, instead i think it runs with par =
=3D 8 (8 core, i=C2=A0 always get 8 output)<br>
<br>
what does that mean?<br>
is that wrong or is it really running with 1 degree of par?<br>
<br>
just a note: I never do any setParallelism() command, i leave it automatica=
l<br>
<br>
thanks<br>
Best<br>
<span><font color=3D"#888888">Michele</font></span></blockquote></div><br><=
/div>
</div></div></blockquote></div><br></div>

--001a11c3d55cbb83bf0519b82b0b--