Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
From: Nico Kruber <nico@data-artisans.com>
To: Evgeny Kincharov <Evgeny_Kincharov@epam.com>
Cc: "user@flink.apache.org" <user@flink.apache.org>
Subject: Re: Running streaming job on every node of cluster
Date: Mon, 27 Feb 2017 18:40:48 +0100
Message-ID: <2854760.on0MsTH7LR@nico-work>
Organization: data Artisans
In-Reply-To: <VI1PR0301MB2429898E93A85CD5C4AC60D298570@VI1PR0301MB2429.eurprd03.prod.outlook.com>
References: <VI1PR0301MB2429898E93A85CD5C4AC60D298570@VI1PR0301MB2429.eurprd03.prod.outlook.com>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="nextPart2817035.7KOZWuXnbK"; micalg="pgp-sha1"; protocol="application/pgp-signature"
archived-at: Mon, 27 Feb 2017 17:41:09 -0000

--nextPart2817035.7KOZWuXnbK
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"

What about setting the parallelism[1] to the total number of slots in your=
=20
cluster?
By default, all parts of your program are put into the same slot sharing=20
group[2] and by setting the parallelism you would have this slot (with your=
=20
whole program) in each parallel slot as well (minus/plus operators that hav=
e=20
lower/higher parallelism), if I understand it correctly.


Nico

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/
parallel.html
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/
datastream_api.html#task-chaining-and-resource-groups

On Monday, 27 February 2017 18:17:53 CET Evgeny Kincharov wrote:
> Thanks for your answer.
> The problem is that both slots are seized in the one node. Of course if t=
his
> node has enough free slots. Another nodes idle. I want to utilize cluster
> resource little bit more. May be the other deployment modes allow it.
>=20
> BR, Evgeny.
>=20
> =D0=9E=D1=82: Nico Kruber<mailto:nico@data-artisans.com>
> =D0=9E=D1=82=D0=BF=D1=80=D0=B0=D0=B2=D0=BB=D0=B5=D0=BD=D0=BE: 27 =D1=84=
=D0=B5=D0=B2=D1=80=D0=B0=D0=BB=D1=8F 2017 =D0=B3. =D0=B2 20:07
> =D0=9A=D0=BE=D0=BC=D1=83: user@flink.apache.org<mailto:user@flink.apache.=
org>
> =D0=9A=D0=BE=D0=BF=D0=B8=D1=8F: Evgeny Kincharov<mailto:Evgeny_Kincharov@=
epam.com>
> =D0=A2=D0=B5=D0=BC=D0=B0: Re: Running streaming job on every node of clus=
ter
>=20
> Hi Evgeny,
> I tried to reproduce your example with the following code, having another
> console listening with "nc -l 12345"
>=20
> env.setParallelism(2);
> env.addSource(new SocketTextStreamFunction("localhost", 12345, " ", 3))
>                                .map(new MapFunction<String, String>() {
>                                                @Override
>                                                public String map(final
> String s) throws Exception { return s; }
 })
>                                .addSink(new DiscardingSink<String>());
>=20
> This way, I do get a source with parallelism 1 and map & sink with
> parallelism
 2 and the whole program accompanying 2 slots as expected. You
> can check in the web interface of your cluster how many slots are taken
> after executing one instance of your program.
>=20
> How do you set your parallelism?
>=20
>=20
> Nico
>=20
> On Monday, 27 February 2017 14:04:21 CET Evgeny Kincharov wrote:
>=20
> > Hi,
> >
> >
> >
> > I have the simplest streaming job, and I want to distribute my job on
> > every
 node of my Flink cluster.
> >
> >
> >
> > Job is simple:
> >
> >
> >
> > source (SocketTextStream) -> map -> sink (AsyncHtttpSink).
> >
> >
> >
> > When I increase parallelism of my job when deploying or directly in cod=
e,
> > no
 effect because source is can't work in parallel. Now I reduce "Tasks
> > Slots" to 1 on ever nodes and deploy my job as many times as nodes in t=
he
> > cluster. It works when I have only one job. If I want deploy another in
> > parallel there is no free slots. I hope more convenient way to do that =
is
> > exists. Thanks.
> >
> >
> >
> > BR,
> > Evgeny
>=20
>=20
>=20


--nextPart2817035.7KOZWuXnbK
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part.
Content-Transfer-Encoding: 7Bit

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQTIh4KsbsNd3l7wd+cg8nJL2uqeWQUCWLRkoAAKCRAg8nJL2uqe
WQv6AKD/wlDgfXKsL4ik0WYl/H6AK+PGywCgwRL6vszY2sh/+QlAW6Rk4QfXRhE=
=y5Ug
-----END PGP SIGNATURE-----

--nextPart2817035.7KOZWuXnbK--