flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com>
Subject RE: Internal buffers supervision and yarn vCPUs
Date Thu, 04 Feb 2016 11:02:28 GMT
Ok thanks !

All that’s left is to wait then.


From: ewenstephan@gmail.com [mailto:ewenstephan@gmail.com] On Behalf Of Stephan Ewen
Sent: jeudi 4 février 2016 11:19
To: user@flink.apache.org
Subject: Re: Internal buffers supervision and yarn vCPUs

Concerning the first question:

What you are looking for is backpressure monitoring. If a task cannot push its data to the
next task, it is backpressured.

This pull request adds a first version of backpressure monitoring: https://github.com/apache/flink/pull/1578

We will try and get it merged soon!

On Thu, Feb 4, 2016 at 11:03 AM, Robert Metzger <rmetzger@apache.org<mailto:rmetzger@apache.org>>
Hi Gwen,

let me answer the second question: There is a JIRA to reintroduce the configuration parameter:
https://issues.apache.org/jira/browse/FLINK-2213. I will try to get a fix for this into the
1.0 release.

I think I removed back then because users were unable to define the number of vcores independently
of the number of slots ... and too many users were running into issues with the yarn scheduler
(containers were not started because there were no CPU resources available anymore).

On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <gwenhael.pasquiers@ericsson.com<mailto:gwenhael.pasquiers@ericsson.com>>

I’ve got two more questions on different topic…

First one :
Is there a way to monitor the buffers status. In order to  find bottleneck in our application
we though it could be usefull to be able to have a look at the different exchange buffers’
status. To know if they are full (or as an example if a mapper had to wait before being able
to push it’s data into the buffer). That way we can know where the bottleneck is.

Second one :
On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc” argument that allowed
to specify the number of vCPU per task manager. We cannot find it anymore. Was it removed
? Is there another way to set the number of vCPU. Or did it became useless ?

Thanks in advance.


View raw message