hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ray Chiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6376) Fix long load times of .jhist file in JobHistoryServer
Date Sat, 20 Jun 2015 21:22:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594819#comment-14594819
] 

Ray Chiang commented on MAPREDUCE-6376:
---------------------------------------

Just a quick follow up.  I wrote up a quick program to convert old .jhist files into the new
binary form and loaded them into a new VM.  The speedups are decent at a high # of tasks,
but closer to 2x (versus the 3x at <50k tasks).

751k Mappers/10 Reducers (json)
- Read Time: 163.8 seconds
- Event Time: 16.6 seconds

751k Mappers/10 Reducers (binary)
- Read Time: 86.7 seconds
- Event Time: 17.4 seconds

404k Mappers/4 Reducers (json)
- Read Time: 93 seconds
- Event Time: 8.9 seconds

404k Mappers/4 Reducers (binary)
- Read Time: 49.1 seconds
- Event Time: 9.2 seconds

I'm probably not going to get to a random event generator this week, but this gives us at
least two more data points.

> Fix long load times of .jhist file in JobHistoryServer
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-6376
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6376
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: jobhistoryserver
>    Affects Versions: 2.7.0
>            Reporter: Ray Chiang
>            Assignee: Ray Chiang
>              Labels: supportability
>         Attachments: MAPREDUCE-6376.001.patch, MAPREDUCE-6376.002.patch, MAPREDUCE-6376.003.patch
>
>
> When you click on a Job link in the JHS Web UI, it loads the .jhist file.  For jobs which
have a large number of tasks, the load time can break UI responsiveness.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message