Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
From: Chen Qin <qinnchen@gmail.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-B24DCC45-49F7-4EF1-B880-4219FC50661E
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)
Date: Fri, 6 Jan 2017 10:27:09 -0800
Subject: Re: Increasing parallelism skews/increases overall job processing time linearly
Message-Id: <86871029-C414-4201-BA45-80022E0D6A7A@gmail.com>
References: <CAP7S8jtwXf6fGxBZii79pjThiYaZe6n=A0ft4L9W3AOKYiUQhw@mail.gmail.com> <CAP7S8juVGhjmhu9Ho0LrvWTh7EfPOqo-ZJRL1UUMWWjdCP6M4w@mail.gmail.com>
In-Reply-To: <CAP7S8juVGhjmhu9Ho0LrvWTh7EfPOqo-ZJRL1UUMWWjdCP6M4w@mail.gmail.com>
To: user@flink.apache.org
archived-at: Fri, 06 Jan 2017 18:27:22 -0000


--Apple-Mail-B24DCC45-49F7-4EF1-B880-4219FC50661E
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Just noticed there are only two partitions per topic. Regardless of how larg=
e parallelism set. Only two of those will get partition assigned at most.

Sent from my iPhone

> On Jan 6, 2017, at 02:40, Chakravarthy varaga <chakravarthyvp@gmail.com> w=
rote:
>=20
> Hi All,
>=20
>     Any updates on this?
>=20
> Best Regards
> CVP
>=20
>> On Thu, Jan 5, 2017 at 1:21 PM, Chakravarthy varaga <chakravarthyvp@gmail=
.com> wrote:
>>=20
>> Hi All,
>>=20
>> I have a job as attached.
>>=20
>> I have a 16 Core blade running RHEL 7. The taskmanager default number of s=
lots is set to 1. The source is a kafka stream and each of the 2 sources(top=
ic) have 2 partitions each.
>> What I notice is that when I deploy a job to run with #parallelism=3D2 th=
e total processing time doubles the time it took when the same job was deplo=
yed with #parallelism=3D1. It linearly increases with the parallelism.
>>=20
>> Since the numberof slots is set to 1 per TM, I would assume that the job w=
ould be processed in parallel in 2 different TMs and that each consumer in e=
ach TM is connected to 1 partition of the topic. This therefore should have k=
ept the overall processing time the same or less !!!
>>=20
>> The co-flatmap connects the 2 streams & uses ValueState (checkpointed in =
FS). I think this is distributed among the TMs. My understanding is that the=
 search of values state could be costly between TMs.  Do you sense something=
 wrong here?
>>=20
>> Best Regards
>> CVP
>>=20
>>=20
>>=20
>>=20
>=20

--Apple-Mail-B24DCC45-49F7-4EF1-B880-4219FC50661E
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D=
utf-8"></head><body dir=3D"auto"><div>Just noticed there are only two partit=
ions per topic. Regardless of how large parallelism set. Only two of those w=
ill get partition assigned at most.<br><br>Sent from my iPhone</div><div><br=
>On Jan 6, 2017, at 02:40, Chakravarthy varaga &lt;<a href=3D"mailto:chakrav=
arthyvp@gmail.com">chakravarthyvp@gmail.com</a>&gt; wrote:<br><br></div><blo=
ckquote type=3D"cite"><div><div dir=3D"ltr"><div><div><div>Hi All,<br><br></=
div>&nbsp;&nbsp;&nbsp; Any updates on this?<br><br></div>Best Regards<br></d=
iv>CVP<br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On=
 Thu, Jan 5, 2017 at 1:21 PM, Chakravarthy varaga <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:chakravarthyvp@gmail.com" target=3D"_blank">chakravarthyvp@gma=
il.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"lt=
r"><div><div><div><div><br></div><div>Hi All,<br><br></div>I have a job as a=
ttached.<br></div><div><br></div>I have a 16 Core blade running RHEL 7. The t=
askmanager default number of slots is set to 1. The source is a kafka stream=
 and each of the 2 sources(topic) have 2 partitions each.<br></div><b>What I=
 notice is that when I deploy a job to run with #parallelism=3D2 the total p=
rocessing time doubles the time it took when the same job was deployed with #=
parallelism=3D1. It linearly increases with the parallelism.<br><br></b></di=
v><div>Since the numberof slots is set to 1 per TM, I would assume that the j=
ob would be processed in parallel in 2 different TMs and that each consumer i=
n each TM is connected to 1 partition of the topic. This therefore should ha=
ve kept the overall processing time the same or less !!!<br><br>The co-flatm=
ap connects the 2 streams &amp; uses ValueState (checkpointed in FS). I thin=
k this is distributed among the TMs. My understanding is that the search of v=
alues state could be costly between TMs.&nbsp; Do you sense something wrong h=
ere?<br><br></div><div>Best Regards<br></div><div>CVP<br></div><div><b><br><=
/b></div><b><br></b><div><br><div><br></div></div></div>
</blockquote></div><br></div>
</div></blockquote></body></html>=

--Apple-Mail-B24DCC45-49F7-4EF1-B880-4219FC50661E--