flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: akka timeout
Date Fri, 25 Aug 2017 21:09:57 GMT
Hi,
are you using the RocksDB state backend already?
Maybe writing the state to disk would actually reduce the pressure on the
GC (but of course it'll also reduce throughput a bit).

Are there any known issues with the network? Maybe the network bursts on
restart cause the timeouts?

On Fri, Aug 25, 2017 at 6:17 PM, Steven Wu <stevenz3wu@gmail.com> wrote:

> Bowen,
>
> Heap size is ~50G. CPU was actually pretty low (like <20%) when high GC
> pause and akka timeout was happening. So maybe memory allocation and GC
> wasn't really an issue. I also recently learned that JVM can pause for
> writing to GC log for disk I/O. that is another lead I am pursuing.
>
> Thanks,
> Steven
>
> On Wed, Aug 23, 2017 at 10:58 AM, Bowen Li <bowen.li@offerupnow.com>
> wrote:
>
>> Hi Steven,
>>     Yes, GC is a big overhead, it may cause your CPU utilization to reach
>> 100%, and every process stopped working. We ran into this a while too.
>>
>>     How much memory did you assign to TaskManager? How much the your CPU
>> utilization when your taskmanager is considered 'killed'?
>>
>> Bowen
>>
>>
>>
>> On Wed, Aug 23, 2017 at 10:01 AM, Steven Wu <stevenz3wu@gmail.com> wrote:
>>
>>> Till,
>>>
>>> Once our job was restarted for some reason (e.g. taskmangaer container
>>> got killed), it can stuck in continuous restart loop for hours. Right now,
>>> I suspect it is caused by GC pause during restart, our job has very high
>>> memory allocation in steady state. High GC pause then caused akka timeout,
>>> which then caused jobmanager to think taksmanager containers are
>>> unhealthy/dead and kill them. And the cycle repeats...
>>>
>>> But I hasn't been able to prove or disprove it yet. When I was asking
>>> the question, I was still sifting through metrics and error logs.
>>>
>>> Thanks,
>>> Steven
>>>
>>>
>>> On Tue, Aug 22, 2017 at 1:21 AM, Till Rohrmann <till.rohrmann@gmail.com>
>>> wrote:
>>>
>>>> Hi Steven,
>>>>
>>>> quick correction for Flink 1.2. Indeed the MetricFetcher does not pick
>>>> up the right timeout value from the configuration. Instead it uses a
>>>> hardcoded 10s timeout. This has only been changed recently and is already
>>>> committed in the master. So with the next release 1.4 it will properly pick
>>>> up the right timeout settings.
>>>>
>>>> Just out of curiosity, what's the instability issue you're observing?
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Fri, Aug 18, 2017 at 7:07 PM, Steven Wu <stevenz3wu@gmail.com>
>>>> wrote:
>>>>
>>>>> Till/Chesnay, thanks for the answers. Look like this is a
>>>>> result/symptom of underline stability issue that I am trying to track
down.
>>>>>
>>>>> It is Flink 1.2.
>>>>>
>>>>> On Fri, Aug 18, 2017 at 12:24 AM, Chesnay Schepler <chesnay@apache.org
>>>>> > wrote:
>>>>>
>>>>>> The MetricFetcher always use the default akka timeout value.
>>>>>>
>>>>>>
>>>>>> On 18.08.2017 09:07, Till Rohrmann wrote:
>>>>>>
>>>>>> Hi Steven,
>>>>>>
>>>>>> I thought that the MetricFetcher picks up the right timeout from
the
>>>>>> configuration. Which version of Flink are you using?
>>>>>>
>>>>>> The timeout is not a critical problem for the job health.
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On Fri, Aug 18, 2017 at 7:22 AM, Steven Wu <stevenz3wu@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> We have set akka.ask.timeout to 60 s in yaml file. I also confirmed
>>>>>>> the setting in Flink UI. But I saw akka timeout of 10 s for metric
query
>>>>>>> service. two questions
>>>>>>> 1) why doesn't metric query use the 60 s value configured in
yaml
>>>>>>> file? does it always use default 10 s value?
>>>>>>> 2) could this cause heartbeat failure between task manager and
job
>>>>>>> manager? or is this jut non-critical failure that won't affect
job health?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Steven
>>>>>>>
>>>>>>> 2017-08-17 23:34:33,421 WARN org.apache.flink.runtime.webmonitor.metrics.MetricFetcher
>>>>>>> - Fetching metrics failed. akka.pattern.AskTimeoutException:
Ask
>>>>>>> timed out on [Actor[akka.tcp://flink@1.2.3.4
>>>>>>> :39139/user/MetricQueryService_23cd9db754bb7d123d80e6b1c0be21d6]]
>>>>>>> after [10000 ms] at akka.pattern.PromiseActorRef$$
>>>>>>> anonfun$1.apply$mcV$sp(AskSupport.scala:334) at
>>>>>>> akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) at
>>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
>>>>>>> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>>>>>>> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
>>>>>>> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message