hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7728) Expose and expand container preemptions in Capacity Scheduler queue metrics
Date Fri, 12 Jan 2018 16:03:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324165#comment-16324165
] 

Eric Payne commented on YARN-7728:
----------------------------------

Thanks a lot for the comments, [~sunilg].

bq. n 3.0, we support multiple types, and this covers only cpu and memory. So could we cover
preemption metrics also in case of multi resources.
I agree with this in principle. However, I made a conscious decision not to do this. There
are a couple of difficulties that I see. First, this is not done for other resource metrics
in QueueMetrics (or any of the other system metrics I could find). The resource metrics only
cover memory and vcores. Second, making the metric names match the resource names is a little
difficult if the resource names could be dynamic. Because of these two things, I feel that
solving this should be done all at the same time in a more general JIRA.

{quote}
One more doubt is with aggregateVcoreSecondsPreempted. MutableCounterLong is used for this.
But under one queue, we ll have multiple containers gets preempted and each container resource
size vary drastically. So are we looking for an aggregate resource among all preempted containers
in a given time ?
{quote}
I don't think I understand the question. The metrics are updated when each container is preempted,
and the value keeps increasing over time. Similar to memory, it's basically a metric of total
lost (virtual) cpu cycles due to preemption since the RM was started.

{quote}
 aggregateMegabyteSecondsPreempted: MegaByte seems a bit confusing, MemoryMB is used in another
places as well. Could we use something similar (like prepending memory)
{quote}
Good point. I will update a new patch.

> Expose and expand container preemptions in Capacity Scheduler queue metrics
> ---------------------------------------------------------------------------
>
>                 Key: YARN-7728
>                 URL: https://issues.apache.org/jira/browse/YARN-7728
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.9.0, 2.8.3, 3.0.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: YARN-7728.001.patch
>
>
> YARN-1047 exposed queue metrics for the number of preempted containers to the fair scheduler.
I would like to also expose these to the capacity scheduler and add metrics for the amount
of lost memory seconds and vcore seconds.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message