flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: akka timeout
Date Mon, 25 Sep 2017 07:52:56 GMT
Quick question Steven. Where did you find the documentation concerning that
the death watch interval is linke to the akka ask timeout? It was included
in the past, but I couldn't find it anymore.

Cheers,
Till

On Mon, Sep 25, 2017 at 9:47 AM, Till Rohrmann <trohrmann@apache.org> wrote:

> Great to hear that you could figure things out Steven.
>
> You are right. The death watch is no longer linked to the akka ask
> timeout, because of FLINK-6495. Thanks for the feedback. I will correct the
> documentation.
>
> Cheers,
> Till
>
> On Sat, Sep 23, 2017 at 10:24 AM, Steven Wu <stevenz3wu@gmail.com> wrote:
>
>> just to close the thread. akka death watch was triggered by high GC
>> pause, which is caused by memory leak in our code during Flink job restart.
>>
>> noted that akka.ask.timeout wasn't related to akka death watch, which
>> Flink has documented and linked.
>>
>> On Sat, Aug 26, 2017 at 10:58 AM, Steven Wu <stevenz3wu@gmail.com> wrote:
>>
>>> this is a stateless job. so we don't use RocksDB.
>>>
>>> yeah. network can also be a possibility. will keep it in the radar.
>>> unfortunately, our metrics system don't have the tcp metrics when running
>>> inside containers.
>>>
>>> On Fri, Aug 25, 2017 at 2:09 PM, Robert Metzger <rmetzger@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>> are you using the RocksDB state backend already?
>>>> Maybe writing the state to disk would actually reduce the pressure on
>>>> the GC (but of course it'll also reduce throughput a bit).
>>>>
>>>> Are there any known issues with the network? Maybe the network bursts
>>>> on restart cause the timeouts?
>>>>
>>>>
>>>> On Fri, Aug 25, 2017 at 6:17 PM, Steven Wu <stevenz3wu@gmail.com>
>>>> wrote:
>>>>
>>>>> Bowen,
>>>>>
>>>>> Heap size is ~50G. CPU was actually pretty low (like <20%) when high
>>>>> GC pause and akka timeout was happening. So maybe memory allocation and
GC
>>>>> wasn't really an issue. I also recently learned that JVM can pause for
>>>>> writing to GC log for disk I/O. that is another lead I am pursuing.
>>>>>
>>>>> Thanks,
>>>>> Steven
>>>>>
>>>>> On Wed, Aug 23, 2017 at 10:58 AM, Bowen Li <bowen.li@offerupnow.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Steven,
>>>>>>     Yes, GC is a big overhead, it may cause your CPU utilization
to
>>>>>> reach 100%, and every process stopped working. We ran into this a
while too.
>>>>>>
>>>>>>     How much memory did you assign to TaskManager? How much the your
>>>>>> CPU utilization when your taskmanager is considered 'killed'?
>>>>>>
>>>>>> Bowen
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 23, 2017 at 10:01 AM, Steven Wu <stevenz3wu@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Till,
>>>>>>>
>>>>>>> Once our job was restarted for some reason (e.g. taskmangaer
>>>>>>> container got killed), it can stuck in continuous restart loop
for hours.
>>>>>>> Right now, I suspect it is caused by GC pause during restart,
our job has
>>>>>>> very high memory allocation in steady state. High GC pause then
caused akka
>>>>>>> timeout, which then caused jobmanager to think taksmanager containers
are
>>>>>>> unhealthy/dead and kill them. And the cycle repeats...
>>>>>>>
>>>>>>> But I hasn't been able to prove or disprove it yet. When I was
>>>>>>> asking the question, I was still sifting through metrics and
error logs.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Steven
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 22, 2017 at 1:21 AM, Till Rohrmann <
>>>>>>> till.rohrmann@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Steven,
>>>>>>>>
>>>>>>>> quick correction for Flink 1.2. Indeed the MetricFetcher
does not
>>>>>>>> pick up the right timeout value from the configuration. Instead
it uses a
>>>>>>>> hardcoded 10s timeout. This has only been changed recently
and is already
>>>>>>>> committed in the master. So with the next release 1.4 it
will properly pick
>>>>>>>> up the right timeout settings.
>>>>>>>>
>>>>>>>> Just out of curiosity, what's the instability issue you're
>>>>>>>> observing?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Till
>>>>>>>>
>>>>>>>> On Fri, Aug 18, 2017 at 7:07 PM, Steven Wu <stevenz3wu@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Till/Chesnay, thanks for the answers. Look like this
is a
>>>>>>>>> result/symptom of underline stability issue that I am
trying to track down.
>>>>>>>>>
>>>>>>>>> It is Flink 1.2.
>>>>>>>>>
>>>>>>>>> On Fri, Aug 18, 2017 at 12:24 AM, Chesnay Schepler <
>>>>>>>>> chesnay@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> The MetricFetcher always use the default akka timeout
value.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 18.08.2017 09:07, Till Rohrmann wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Steven,
>>>>>>>>>>
>>>>>>>>>> I thought that the MetricFetcher picks up the right
timeout from
>>>>>>>>>> the configuration. Which version of Flink are you
using?
>>>>>>>>>>
>>>>>>>>>> The timeout is not a critical problem for the job
health.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Till
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 18, 2017 at 7:22 AM, Steven Wu <stevenz3wu@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We have set akka.ask.timeout to 60 s in yaml
file. I also
>>>>>>>>>>> confirmed the setting in Flink UI. But I saw
akka timeout of 10 s for
>>>>>>>>>>> metric query service. two questions
>>>>>>>>>>> 1) why doesn't metric query use the 60 s value
configured in
>>>>>>>>>>> yaml file? does it always use default 10 s value?
>>>>>>>>>>> 2) could this cause heartbeat failure between
task manager and
>>>>>>>>>>> job manager? or is this jut non-critical failure
that won't affect job
>>>>>>>>>>> health?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Steven
>>>>>>>>>>>
>>>>>>>>>>> 2017-08-17 23:34:33,421 WARN org.apache.flink.runtime.webmonitor.metrics.MetricFetcher
>>>>>>>>>>> - Fetching metrics failed. akka.pattern.AskTimeoutException:
>>>>>>>>>>> Ask timed out on [Actor[akka.tcp://flink@1.2.3.4
>>>>>>>>>>> :39139/user/MetricQueryService_23cd9db754bb7d123d80e6b1c0be21d6]]
>>>>>>>>>>> after [10000 ms] at akka.pattern.PromiseActorRef$$
>>>>>>>>>>> anonfun$1.apply$mcV$sp(AskSupport.scala:334)
at
>>>>>>>>>>> akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
at
>>>>>>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
>>>>>>>>>>> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>>>>>>>>>>> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message