Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
From: Andrew Ge Wu <andrew.ge-wu@eniro.com>
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Cluster failure after zookeeper glitch.
Message-Id: <DC372EB4-88EB-42F9-89AD-0C618E97B90B@eniro.com>
Date: Thu, 19 Jan 2017 13:16:57 +0100
To: user@flink.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 19 Jan 2017 12:17:07 -0000

Hi,


We recently had several zookeeper glitch, when that happens it seems to tak=
e flink cluster with it.

We are running on 1.03

It started like this:


2017-01-19 11:52:13,047 INFO  org.apache.zookeeper.ClientCnxn              =
                 - Unable to read additional data from server sessionid 0x1=
59b505820a0008, likely server has closed socket, closing socket connection =
and attempting reconnect
2017-01-19 11:52:13,047 INFO  org.apache.zookeeper.ClientCnxn              =
                 - Unable to read additional data from server sessionid 0x1=
59b505820a0009, likely server has closed socket, closing socket connection =
and attempting reconnect
2017-01-19 11:52:13,151 INFO  org.apache.flink.shaded.org.apache.curator.fr=
amework.state.ConnectionStateManager  - State change: SUSPENDED
2017-01-19 11:52:13,151 INFO  org.apache.flink.shaded.org.apache.curator.fr=
amework.state.ConnectionStateManager  - State change: SUSPENDED
2017-01-19 11:52:13,166 WARN  org.apache.flink.runtime.jobmanager.ZooKeeper=
SubmittedJobGraphStore  - ZooKeeper connection SUSPENDED. Changes to the su=
bmitted job graphs are not monitored (temporarily).
2017-01-19 11:52:13,169 INFO  org.apache.flink.runtime.jobmanager.JobManage=
r                - JobManager akka://flink/user/jobmanager#1976923422 was r=
evoked leadership.
2017-01-19 11:52:13,179 INFO  org.apache.flink.runtime.executiongraph.Execu=
tionGraph        - op1 -> (Map, Map -> op2) (18/24) (5336dd375eb12616c5a0e9=
3c84f93465) switched from RUNNING to FAILED


Then our web-ui stopped serving and job manager stuck in an exception loop =
like this:
2017-01-19 13:05:13,521 WARN  org.apache.flink.runtime.jobmanager.JobManage=
r                - Discard message LeaderSessionMessage(0318ecf5-7069-41b2-=
a793-2f24bdbaa287,01/19/2017 13:05:13     Job execution switched to status =
RESTARTING.) because the expected leader session I
D None did not equal the received leader session ID Some(0318ecf5-7069-41b2=
-a793-2f24bdbaa287).
2017-01-19 13:05:13,521 INFO  org.apache.flink.runtime.executiongraph.resta=
rt.FixedDelayRestartStrategy  - Delaying retry of job execution for xxxxx m=
s =E2=80=A6


Is it because we misconfigured anything? or this is expected behavior? When=
 this happens we have to restart the cluster to bring it back.


Thanks!


Andrew
--=20
Confidentiality Notice: This e-mail transmission may contain confidential=
=20
or legally privileged information that is intended only for the individual=
=20
or entity named in the e-mail address. If you are not the intended=20
recipient, you are hereby notified that any disclosure, copying,=20
distribution, or reliance upon the contents of this e-mail is strictly=20
prohibited and may be unlawful. If you have received this e-mail in error,=
=20
please notify the sender immediately by return e-mail and delete all copies=
=20
of this message.