flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Fetching metrics failed.
Date Thu, 20 Apr 2017 17:07:57 GMT
Hello,

the MetricQueryService is used by the webUI to fetch fetch metrics from 
the JobManager and all TaskManagers. It is only used when the
webUI is accessed.

Based on the logs you gave the TaskManager isn't killed by the 
JobManager; instead the JobManager only detected that the TaskManager 
has shut down.

It is highly unlikely that the MetricQueryService is the cause of this; 
the exception you are seeing is due to the TaskManager being no longer 
reachable. Can't fetch metrics when the TaskManager isn't there anymore.

How do you mange the Flink cluster? (Yarn etc.) Given that no exception 
appears in the log i would assume that the TaskManager JVM was killed 
from the outside.

Regards,
Chesnay

On 20.04.2017 18:42, Jason Brelloch wrote:
> Hey all,
>
> So we are doing some experimenting around large keyed state in Flink 
> 1.2 on a single task manager and we keep having our task manager 
> killed by the job manager after about 10 minutes due to this exception:
>
> Fetching metrics failed.
> akka.pattern.AskTimeoutException: Ask timed out on 
> [Actor[akka.tcp://flink@flink-s-load-uscen-a-c001-n011:37244/user/MetricQueryService_0f7bba0b16b18e83b69c4a50e657bb1f]]

> after [10000 ms]
>
> The task manager logs show nothing out of the ordinary, but the job 
> manager logs shows this:
>
> 2017-04-19 20:56:52,230 Association with remote system 
> [akka.tcp://flink@flink-s-load-uscen-a-c001-n011:37244] has failed, 
> address is now gated for [5000] ms. Reason: [Disassociated]
> 2017-04-19 20:56:53,986 Fetching metrics failed.
> 2017-04-19 20:57:43,584 Association with remote system 
> [akka.tcp://flink@flink-s-load-uscen-a-c001-n011:37244] has failed, 
> address is now gated for [5000] ms. Reason: [Association failed with 
> [akka.tcp://flink@flink-s-load-uscen-a-c001-n011:37244]] Caused by: 
> [Connection refused: flink-s-load-uscen-a-c001-n011/10.34.48.40:37244 
> <http://10.34.48.40:37244>]
> 2017-04-19 20:57:49,517 Detected unreachable: 
> [akka.tcp://flink@flink-s-load-uscen-a-c001-n011:37244]
> 2017-04-19 20:57:49,517 Task manager 
> akka.tcp://flink@flink-s-load-uscen-a-c001-n011:37244/user/taskmanager 
> terminated.
>
> The weird part is, we have not set up any metrics reporters or 
> anything so I am not really sure why the Job Manager is asking the 
> task manager about them.  Is there a way to disable these metrics 
> requests, or does anyone know what is causing them?
>
> Thanks,
> -- 
> *Jason Brelloch* | Product Developer
> 3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305
> <http://www.bettercloud.com/>
> Subscribe to the BetterCloud Monitor 
> <https://www.bettercloud.com/monitor?utm_source=bettercloud_email&utm_medium=email_signature&utm_campaign=monitor_launch>
- 
> Get IT delivered to your inbox



Mime
View raw message