flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Internal buffers supervision and yarn vCPUs
Date Thu, 04 Feb 2016 10:19:19 GMT
Concerning the first question:

What you are looking for is backpressure monitoring. If a task cannot push
its data to the next task, it is backpressured.

This pull request adds a first version of backpressure monitoring:
https://github.com/apache/flink/pull/1578

We will try and get it merged soon!


On Thu, Feb 4, 2016 at 11:03 AM, Robert Metzger <rmetzger@apache.org> wrote:

> Hi Gwen,
>
> let me answer the second question: There is a JIRA to reintroduce the
> configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213.
> I will try to get a fix for this into the 1.0 release.
>
> I think I removed back then because users were unable to define the number
> of vcores independently of the number of slots ... and too many users were
> running into issues with the yarn scheduler (containers were not started
> because there were no CPU resources available anymore).
>
>
>
> On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <
> gwenhael.pasquiers@ericsson.com> wrote:
>
>> Hi,
>>
>>
>>
>> I’ve got two more questions on different topic…
>>
>>
>>
>> First one :
>>
>> Is there a way to monitor the buffers status. In order to  find
>> bottleneck in our application we though it could be usefull to be able to
>> have a look at the different exchange buffers’ status. To know if they are
>> full (or as an example if a mapper had to wait before being able to push
>> it’s data into the buffer). That way we can know where the bottleneck is.
>>
>>
>>
>> Second one :
>>
>> On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc”
>> argument that allowed to specify the number of vCPU per task manager. We
>> cannot find it anymore. Was it removed ? Is there another way to set the
>> number of vCPU. Or did it became useless ?
>>
>>
>>
>> Thanks in advance.
>>
>>
>>
>> Gwen’
>>
>
>

Mime
View raw message