hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
Date Wed, 28 Jan 2015 18:37:37 GMT

    [ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295583#comment-14295583

Wangda Tan commented on YARN-3098:

Hi [~sunilg],
Thanks for thinking about this,.

I think two potential issues we need to take care regarding locks:
1) Deadlock caused by the new introduced locks.
2) Data inconsistency.

For 1),
I think it will be no problem if *a.* we don't try to acquire other locks from methods of
QueueCapacities/ResourceUsage. And *b.* methods of QC/RU are pretty simple, all can terminate
in finite time. 
If a/b meet at the same time. No new deadlock will happen. Agree?

For 2),
Data inconsistency also has two different types,
2.1 Data inconsistency while editing
When we edit data within QC/RU, it still under queue's lock. Which has no data inconsistency

2.2 Data inconsistency while reading
When we read data in QC/RU from outside, it is possible QC in state1 (e.g. container-1 allocated)
but RU in state2 (e.g. container-2 allocated). In the past we have same issue. We cannot avoid
this except we wrap such data to a structure like "QueueStatusInfo" like AppReport.


> Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels
of queues
> -------------------------------------------------------------------------------------------------
>                 Key: YARN-3098
>                 URL: https://issues.apache.org/jira/browse/YARN-3098
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch
> Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to
track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity,
etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability
and fine-grained locking.

This message was sent by Atlassian JIRA

View raw message