Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Saliya Ekanayake <esaliya@gmail.com>
Date: Thu, 7 Jul 2016 23:57:31 -0400
Message-ID: <CA+ssbKVVYTjfMWte56HXwsrJguLqOHMBpE4R=dirkTDreVg0=A@mail.gmail.com>
Subject: Flink is Unstable when TM > 1
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11395e0c545195053717cd4d
archived-at: Fri, 08 Jul 2016 03:57:38 -0000

--001a11395e0c545195053717cd4d
Content-Type: text/plain; charset=UTF-8

Hi,

I've been trying to run the provided KMeans example on a 16 node cluster. I
was testing with 2 Task Managers (TM) per node because each node has 2
sockets (CPUs). A socket contains 12 cores, so I've set the number of slots
per TM as 12.The total parallelism is 384 (12 slots x 2 TMs x 16 nodes).

However, Flink TMs keep failing time to time causing KMeans to fail. The
only explanation I could find from logs is that TMs unregister from Job
Manager. I've increased Akka timeout to 1000s as well.

Any suggestions on this?

The data sizes I tried were 10k points, 250k points, and 1mil points.
Number of centers were 100 to 1000. None of these sizes completed.

Thank you,
Saliya

-- 
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington

--001a11395e0c545195053717cd4d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I&#39;ve been trying to run the pro=
vided KMeans example on a 16 node cluster. I was testing with 2 Task Manage=
rs (TM) per node because each node has 2 sockets (CPUs). A socket contains =
12 cores, so I&#39;ve set the number of slots per TM as 12.The total parall=
elism is 384 (12 slots x 2 TMs x 16 nodes).=C2=A0</div><div><br></div><div>=
However, Flink TMs keep failing time to time causing KMeans to fail. The on=
ly explanation I could find from logs is that TMs unregister from Job Manag=
er. I&#39;ve increased Akka timeout to 1000s as well.=C2=A0</div><div><br><=
/div><div>Any suggestions on this?</div><div><br></div><div>The data sizes =
I tried were 10k points, 250k points, and 1mil points. Number of centers we=
re 100 to 1000. None of these sizes completed.<br></div><div><br></div><div=
>Thank you,<br>Saliya</div><div><div><br></div>-- <br><div class=3D"gmail_s=
ignature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=
=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><span style=3D"color:rgb(136,136=
,136)">Saliya Ekanayake</span></div><div dir=3D"ltr">Ph.D. Candidate | Rese=
arch Assistant</div><div dir=3D"ltr">School of Informatics and Computing | =
Digital Science Center</div><div dir=3D"ltr">Indiana University, Bloomingto=
n<br><br><font color=3D"#888888"></font></div></div></div></div></div></div=
>
</div></div>

--001a11395e0c545195053717cd4d--