hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4091) Add REST API to retrieve scheduler activity
Date Fri, 29 Jul 2016 15:20:20 GMT

     [ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sunil G updated YARN-4091:
--------------------------
    Attachment: SchedulerActivityManager-TestReport.pdf

HI [~ChenGe] and [~leftnoteasy]

I got some time to do test  with this patch. I thought of sharing test results here along
with few inputs.

I added this comments in the doc as well.

Comments:
# I think Diagnostic message could be improved.  "do not need more resource" => “Applications
does not need more resource”
# For node activity, "priority": "-1" does not make sense. Could we hide the same from node
level and show for app (container)?
# timeStamp is not meaningful ("timeStamp": "1469792611186"). Its could be date and time or
relative to previous activity.
# *finalAllocationState* is one of the entry for application. Could we say *finalAppAllocationState*.
# In queue level, is “allocationState” meaningful? I think we can hide in queue level,
thoughts.?
# As mentioned earlier, priority could be hidden in places where its -1.
# As an improvement, its better to give pending resource requests per app after allocation.
So we can get some idea and can help a lot.
# when I tested below test case "allocation for an application is done and app is running.
Second app is awaiting due to AM resource percentage." I could not get expected result. Am
I missing something.? Test case 6 in the report.
# Could we also print node_label too when container is allocated


I tried some more cases and will try enhancing this report.



> Add REST API to retrieve scheduler activity
> -------------------------------------------
>
>                 Key: YARN-4091
>                 URL: https://issues.apache.org/jira/browse/YARN-4091
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler, resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Sunil G
>            Assignee: Chen Ge
>         Attachments: Improvement on debugdiagnostic information - YARN.pdf, SchedulerActivityManager-TestReport.pdf,
YARN-4091-design-doc-v1.pdf, YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch,
YARN-4091.preliminary.1.patch, app_activities.json, node_activities.json
>
>
> As schedulers are improved with various new capabilities, more configurations which tunes
the schedulers starts to take actions such as limit assigning containers to an application,
or introduce delay to allocate container etc. 
> There are no clear information passed down from scheduler to outerworld under these various
scenarios. This makes debugging very tougher.
> This ticket is an effort to introduce more defined states on various parts in scheduler
where it skips/rejects container assignment, activate application etc. Such information will
help user to know whats happening in scheduler.
> Attaching a short proposal for initial discussion. We would like to improve on this as
we discuss.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message