flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: akka timeout
Date Mon, 25 Sep 2017 07:47:35 GMT
Great to hear that you could figure things out Steven.

You are right. The death watch is no longer linked to the akka ask timeout,
because of FLINK-6495. Thanks for the feedback. I will correct the
documentation.

Cheers,
Till

On Sat, Sep 23, 2017 at 10:24 AM, Steven Wu <stevenz3wu@gmail.com> wrote:

> just to close the thread. akka death watch was triggered by high GC pause,
> which is caused by memory leak in our code during Flink job restart.
>
> noted that akka.ask.timeout wasn't related to akka death watch, which
> Flink has documented and linked.
>
> On Sat, Aug 26, 2017 at 10:58 AM, Steven Wu <stevenz3wu@gmail.com> wrote:
>
>> this is a stateless job. so we don't use RocksDB.
>>
>> yeah. network can also be a possibility. will keep it in the radar.
>> unfortunately, our metrics system don't have the tcp metrics when running
>> inside containers.
>>
>> On Fri, Aug 25, 2017 at 2:09 PM, Robert Metzger <rmetzger@apache.org>
>> wrote:
>>
>>> Hi,
>>> are you using the RocksDB state backend already?
>>> Maybe writing the state to disk would actually reduce the pressure on
>>> the GC (but of course it'll also reduce throughput a bit).
>>>
>>> Are there any known issues with the network? Maybe the network bursts on
>>> restart cause the timeouts?
>>>
>>>
>>> On Fri, Aug 25, 2017 at 6:17 PM, Steven Wu <stevenz3wu@gmail.com> wrote:
>>>
>>>> Bowen,
>>>>
>>>> Heap size is ~50G. CPU was actually pretty low (like <20%) when high GC
>>>> pause and akka timeout was happening. So maybe memory allocation and GC
>>>> wasn't really an issue. I also recently learned that JVM can pause for
>>>> writing to GC log for disk I/O. that is another lead I am pursuing.
>>>>
>>>> Thanks,
>>>> Steven
>>>>
>>>> On Wed, Aug 23, 2017 at 10:58 AM, Bowen Li <bowen.li@offerupnow.com>
>>>> wrote:
>>>>
>>>>> Hi Steven,
>>>>>     Yes, GC is a big overhead, it may cause your CPU utilization to
>>>>> reach 100%, and every process stopped working. We ran into this a while
too.
>>>>>
>>>>>     How much memory did you assign to TaskManager? How much the your
>>>>> CPU utilization when your taskmanager is considered 'killed'?
>>>>>
>>>>> Bowen
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 23, 2017 at 10:01 AM, Steven Wu <stevenz3wu@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Till,
>>>>>>
>>>>>> Once our job was restarted for some reason (e.g. taskmangaer
>>>>>> container got killed), it can stuck in continuous restart loop for
hours.
>>>>>> Right now, I suspect it is caused by GC pause during restart, our
job has
>>>>>> very high memory allocation in steady state. High GC pause then caused
akka
>>>>>> timeout, which then caused jobmanager to think taksmanager containers
are
>>>>>> unhealthy/dead and kill them. And the cycle repeats...
>>>>>>
>>>>>> But I hasn't been able to prove or disprove it yet. When I was asking
>>>>>> the question, I was still sifting through metrics and error logs.
>>>>>>
>>>>>> Thanks,
>>>>>> Steven
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 22, 2017 at 1:21 AM, Till Rohrmann <
>>>>>> till.rohrmann@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Steven,
>>>>>>>
>>>>>>> quick correction for Flink 1.2. Indeed the MetricFetcher does
not
>>>>>>> pick up the right timeout value from the configuration. Instead
it uses a
>>>>>>> hardcoded 10s timeout. This has only been changed recently and
is already
>>>>>>> committed in the master. So with the next release 1.4 it will
properly pick
>>>>>>> up the right timeout settings.
>>>>>>>
>>>>>>> Just out of curiosity, what's the instability issue you're observing?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Fri, Aug 18, 2017 at 7:07 PM, Steven Wu <stevenz3wu@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Till/Chesnay, thanks for the answers. Look like this is a
>>>>>>>> result/symptom of underline stability issue that I am trying
to track down.
>>>>>>>>
>>>>>>>> It is Flink 1.2.
>>>>>>>>
>>>>>>>> On Fri, Aug 18, 2017 at 12:24 AM, Chesnay Schepler <
>>>>>>>> chesnay@apache.org> wrote:
>>>>>>>>
>>>>>>>>> The MetricFetcher always use the default akka timeout
value.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 18.08.2017 09:07, Till Rohrmann wrote:
>>>>>>>>>
>>>>>>>>> Hi Steven,
>>>>>>>>>
>>>>>>>>> I thought that the MetricFetcher picks up the right timeout
from
>>>>>>>>> the configuration. Which version of Flink are you using?
>>>>>>>>>
>>>>>>>>> The timeout is not a critical problem for the job health.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Till
>>>>>>>>>
>>>>>>>>> On Fri, Aug 18, 2017 at 7:22 AM, Steven Wu <stevenz3wu@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We have set akka.ask.timeout to 60 s in yaml file.
I also
>>>>>>>>>> confirmed the setting in Flink UI. But I saw akka
timeout of 10 s for
>>>>>>>>>> metric query service. two questions
>>>>>>>>>> 1) why doesn't metric query use the 60 s value configured
in yaml
>>>>>>>>>> file? does it always use default 10 s value?
>>>>>>>>>> 2) could this cause heartbeat failure between task
manager and
>>>>>>>>>> job manager? or is this jut non-critical failure
that won't affect job
>>>>>>>>>> health?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Steven
>>>>>>>>>>
>>>>>>>>>> 2017-08-17 23:34:33,421 WARN org.apache.flink.runtime.webmonitor.metrics.MetricFetcher
>>>>>>>>>> - Fetching metrics failed. akka.pattern.AskTimeoutException:
Ask
>>>>>>>>>> timed out on [Actor[akka.tcp://flink@1.2.3.4
>>>>>>>>>> :39139/user/MetricQueryService_23cd9db754bb7d123d80e6b1c0be21d6]]
>>>>>>>>>> after [10000 ms] at akka.pattern.PromiseActorRef$$
>>>>>>>>>> anonfun$1.apply$mcV$sp(AskSupport.scala:334) at
>>>>>>>>>> akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
at
>>>>>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
>>>>>>>>>> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>>>>>>>>>> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message