flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <stevenz...@gmail.com>
Subject Re: akka timeout
Date Tue, 26 Sep 2017 07:55:53 GMT
Till, sorry for the confusion. I meant Flink documentation has the correct
info. our code was mistakenly referring to akka.ask.timeout for death watch.

On Mon, Sep 25, 2017 at 3:52 PM, Till Rohrmann <trohrmann@apache.org> wrote:

> Quick question Steven. Where did you find the documentation concerning
> that the death watch interval is linke to the akka ask timeout? It was
> included in the past, but I couldn't find it anymore.
>
> Cheers,
> Till
>
> On Mon, Sep 25, 2017 at 9:47 AM, Till Rohrmann <trohrmann@apache.org>
> wrote:
>
>> Great to hear that you could figure things out Steven.
>>
>> You are right. The death watch is no longer linked to the akka ask
>> timeout, because of FLINK-6495. Thanks for the feedback. I will correct the
>> documentation.
>>
>> Cheers,
>> Till
>>
>> On Sat, Sep 23, 2017 at 10:24 AM, Steven Wu <stevenz3wu@gmail.com> wrote:
>>
>>> just to close the thread. akka death watch was triggered by high GC
>>> pause, which is caused by memory leak in our code during Flink job restart.
>>>
>>> noted that akka.ask.timeout wasn't related to akka death watch, which
>>> Flink has documented and linked.
>>>
>>> On Sat, Aug 26, 2017 at 10:58 AM, Steven Wu <stevenz3wu@gmail.com>
>>> wrote:
>>>
>>>> this is a stateless job. so we don't use RocksDB.
>>>>
>>>> yeah. network can also be a possibility. will keep it in the radar.
>>>> unfortunately, our metrics system don't have the tcp metrics when running
>>>> inside containers.
>>>>
>>>> On Fri, Aug 25, 2017 at 2:09 PM, Robert Metzger <rmetzger@apache.org>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> are you using the RocksDB state backend already?
>>>>> Maybe writing the state to disk would actually reduce the pressure on
>>>>> the GC (but of course it'll also reduce throughput a bit).
>>>>>
>>>>> Are there any known issues with the network? Maybe the network bursts
>>>>> on restart cause the timeouts?
>>>>>
>>>>>
>>>>> On Fri, Aug 25, 2017 at 6:17 PM, Steven Wu <stevenz3wu@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Bowen,
>>>>>>
>>>>>> Heap size is ~50G. CPU was actually pretty low (like <20%) when
high
>>>>>> GC pause and akka timeout was happening. So maybe memory allocation
and GC
>>>>>> wasn't really an issue. I also recently learned that JVM can pause
for
>>>>>> writing to GC log for disk I/O. that is another lead I am pursuing.
>>>>>>
>>>>>> Thanks,
>>>>>> Steven
>>>>>>
>>>>>> On Wed, Aug 23, 2017 at 10:58 AM, Bowen Li <bowen.li@offerupnow.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Steven,
>>>>>>>     Yes, GC is a big overhead, it may cause your CPU utilization
to
>>>>>>> reach 100%, and every process stopped working. We ran into this
a while too.
>>>>>>>
>>>>>>>     How much memory did you assign to TaskManager? How much the
your
>>>>>>> CPU utilization when your taskmanager is considered 'killed'?
>>>>>>>
>>>>>>> Bowen
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 23, 2017 at 10:01 AM, Steven Wu <stevenz3wu@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Till,
>>>>>>>>
>>>>>>>> Once our job was restarted for some reason (e.g. taskmangaer
>>>>>>>> container got killed), it can stuck in continuous restart
loop for hours.
>>>>>>>> Right now, I suspect it is caused by GC pause during restart,
our job has
>>>>>>>> very high memory allocation in steady state. High GC pause
then caused akka
>>>>>>>> timeout, which then caused jobmanager to think taksmanager
containers are
>>>>>>>> unhealthy/dead and kill them. And the cycle repeats...
>>>>>>>>
>>>>>>>> But I hasn't been able to prove or disprove it yet. When
I was
>>>>>>>> asking the question, I was still sifting through metrics
and error logs.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Steven
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 22, 2017 at 1:21 AM, Till Rohrmann <
>>>>>>>> till.rohrmann@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Steven,
>>>>>>>>>
>>>>>>>>> quick correction for Flink 1.2. Indeed the MetricFetcher
does not
>>>>>>>>> pick up the right timeout value from the configuration.
Instead it uses a
>>>>>>>>> hardcoded 10s timeout. This has only been changed recently
and is already
>>>>>>>>> committed in the master. So with the next release 1.4
it will properly pick
>>>>>>>>> up the right timeout settings.
>>>>>>>>>
>>>>>>>>> Just out of curiosity, what's the instability issue you're
>>>>>>>>> observing?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Till
>>>>>>>>>
>>>>>>>>> On Fri, Aug 18, 2017 at 7:07 PM, Steven Wu <stevenz3wu@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Till/Chesnay, thanks for the answers. Look like this
is a
>>>>>>>>>> result/symptom of underline stability issue that
I am trying to track down.
>>>>>>>>>>
>>>>>>>>>> It is Flink 1.2.
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 18, 2017 at 12:24 AM, Chesnay Schepler
<
>>>>>>>>>> chesnay@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> The MetricFetcher always use the default akka
timeout value.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 18.08.2017 09:07, Till Rohrmann wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Steven,
>>>>>>>>>>>
>>>>>>>>>>> I thought that the MetricFetcher picks up the
right timeout from
>>>>>>>>>>> the configuration. Which version of Flink are
you using?
>>>>>>>>>>>
>>>>>>>>>>> The timeout is not a critical problem for the
job health.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Till
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 18, 2017 at 7:22 AM, Steven Wu <stevenz3wu@gmail.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We have set akka.ask.timeout to 60 s in yaml
file. I also
>>>>>>>>>>>> confirmed the setting in Flink UI. But I
saw akka timeout of 10 s for
>>>>>>>>>>>> metric query service. two questions
>>>>>>>>>>>> 1) why doesn't metric query use the 60 s
value configured in
>>>>>>>>>>>> yaml file? does it always use default 10
s value?
>>>>>>>>>>>> 2) could this cause heartbeat failure between
task manager and
>>>>>>>>>>>> job manager? or is this jut non-critical
failure that won't affect job
>>>>>>>>>>>> health?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Steven
>>>>>>>>>>>>
>>>>>>>>>>>> 2017-08-17 23:34:33,421 WARN org.apache.flink.runtime.webmonitor.metrics.MetricFetcher
>>>>>>>>>>>> - Fetching metrics failed. akka.pattern.AskTimeoutException:
>>>>>>>>>>>> Ask timed out on [Actor[akka.tcp://flink@1.2.3.4
>>>>>>>>>>>> :39139/user/MetricQueryService_23cd9db754bb7d123d80e6b1c0be21d6]]
>>>>>>>>>>>> after [10000 ms] at akka.pattern.PromiseActorRef$$
>>>>>>>>>>>> anonfun$1.apply$mcV$sp(AskSupport.scala:334)
at
>>>>>>>>>>>> akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
at
>>>>>>>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
>>>>>>>>>>>> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>>>>>>>>>>>> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
>>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>>>>>>>>>>>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message