hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3309) Report the AM of an application in the UI
Date Mon, 14 Nov 2011 17:11:52 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149751#comment-13149751
] 

jiraposter@reviews.apache.org commented on MAPREDUCE-3309:
----------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2822/
-----------------------------------------------------------

Review request for Tom Graves, Robert Evans and Mark Holderbaugh.


Summary
-------

Make provision to report the AM hostname of an application in the RM/JHS UI.
It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history.
Digging through the logs is an option but since there is no consistency maintained in naming
of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder.

------------------
The big idea is to add the application master to the jobhistory ui so that can be sort and
increase search capabilities. Currently, adding anything to the jobhistory apps page means
to add to the job history file name since that is parsed to get job meta data. This is done
for performance reasons to prevent reading all jobhistory files.


This addresses bug MAPREDUCE-3309.
    http://issues.apache.org/jira/browse/MAPREDUCE-3309


Diffs
-----

  hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java
c4ef938 
  hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
a27ba15 
  hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
8db4caf 
  hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
700fdb3 
  hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
7c175d1 
  hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
1e9215f 
  hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsList.java
57e695c 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
ca4ab18 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
d412a63 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
0a6b969 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsView.java
ddb904d 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
3c0f3bb 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
3a35045 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
0d0271c 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
ae886cf 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
4abcd34 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
dd19ed0 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
5a67576 
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
f82c1d5 

Diff: https://reviews.apache.org/r/2822/diff


Testing
-------

Manual test done:

1. Verify correct application master on job history apps page with jobhistory files using
new format
2. Verify old job history files are still usable to retrieve meta data minus the application
master
3. Verify all fields are searchable using field search as well and general search


Thanks,

Jonathan


                
> Report the AM of an application in the UI
> -----------------------------------------
>
>                 Key: MAPREDUCE-3309
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Ramya Sunil
>            Assignee: Jonathan Eagles
>         Attachments: MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch,
MAPREDUCE-3309.patch
>
>
> Make provision to report the AM hostname of an application in the RM/JHS UI. 
> It is difficult to trace back the AM on which an app ran when there are 100+ jobs in
history. Digging through the logs is an option but since there is no consistency maintained
in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message