Mailing-List: contact dev-help@flink.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of ewenstephan@gmail.com
 designates 209.85.220.181 as permitted sender)
MIME-Version: 1.0
Sender: ewenstephan@gmail.com
In-Reply-To: 
 <979DB9666496EA4AA881B4E447D3040001B2890E@MXMA2012.hpi.uni-potsdam.de>
References: 
 <979DB9666496EA4AA881B4E447D3040001B2846C@MXMA2012.hpi.uni-potsdam.de>
	<CANC1h_uoQHzZ09bGBrKg+pEiDvrz2PnMP=fk2LcTtzAyMbvXVw@mail.gmail.com>
	<979DB9666496EA4AA881B4E447D3040001B2890E@MXMA2012.hpi.uni-potsdam.de>
Date: Tue, 18 Nov 2014 14:07:31 +0100
Message-ID: 
 <CANC1h_sG-KQdiePKHcT3v=1S5osN828XxuoTOe6iOnvxadfn=A@mail.gmail.com>
Subject: Re: Heartbeat lost
From: Stephan Ewen <sewen@apache.org>
To: dev@flink.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c2487a2747f7050821c781

--001a11c2487a2747f7050821c781
Content-Type: text/plain; charset=UTF-8

The heartbeats currently go through the RPC service which is soon to be
replaced by akka. So any fix there would be temporary.

You can try increasing the thread priority, let us know if it works.

Otherwise you can increase the heart beat timeout via
"jobmanager.max-heartbeat-delay-before-failure.sec". CAREFUL: The keys says
seconds, but the value is in milliseconds. We actually need to fix that

Stephan


On Tue, Nov 18, 2014 at 1:25 PM, Kruse, Sebastian <Sebastian.Kruse@hpi.de>
wrote:

> I am using the RemoteCollectorOutputFormat (if you recall, Fabian
> Tschirschnitz contributed this) to send the output data to the driver which
> happens to run on the same machine as the jobmanager. In some cases, this
> output becomes huge, I assume this to be the problem.
>
> However, since the heartbeat runs in its own thread, we could assign it a
> higher priority than regular driver/jobmanager code, to avoid the
> suppression of heartbeats. Or do I miss something?
>
> Cheers,
> Sebastian
>
> -----Original Message-----
> From: ewenstephan@gmail.com [mailto:ewenstephan@gmail.com] On Behalf Of
> Stephan Ewen
> Sent: Dienstag, 18. November 2014 10:57
> To: dev@flink.incubator.apache.org
> Subject: Re: Heartbeat lost
>
> Yes, that sounds like a good idea.
>
> I have experienced that occasionally before, under high parallelism and
> algorithms where the task manager got long garbage collection stalls...
>
> The default timeout (30 seconds) can be aggressive for sich jobs...
>
> Stephan
> Am 18.11.2014 09:47 schrieb "Kruse, Sebastian" <Sebastian.Kruse@hpi.de>:
>
> > Hi everyone,
> >
> > In some of my jobs, I occasionally encounter the problem, that some of
> > the task managers lose the heartbeat connection to the job manager.
> > The jobmanager did not crash, though. Here an excerpt from the dashboard:
> >
> > Error: java.lang.Exception: TaskManager lost heartbeat connection to
> > JobManager at
> > org.apache.flink.runtime.taskmanager.TaskManager.registerAndRunHeartbe
> > atLoop(TaskManager.java:847)
> > at
> > org.apache.flink.runtime.taskmanager.TaskManager.access$000(TaskManage
> > r.java:109)
> > at
> > org.apache.flink.runtime.taskmanager.TaskManager$1.run(TaskManager.jav
> > a:365)
> >
> > I am not sure if this is a bug. I rather figure that the network or
> > jobmanager workload is too high, so that somehow the heartbeats do not
> > arrive (on time), but that's a mere guess. A first step for me could
> > be to increase the heartbeat interval.
> >
> > Has anyone of you encountered this problem or do you have any ideas on
> > how to avoid this issue?
> >
> > Thanks,
> > Sebastian
> >
>

--001a11c2487a2747f7050821c781--