hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6394) Speed up Task processing loop in HsTasksBlock#render()
Date Wed, 29 Jul 2015 16:11:06 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646327#comment-14646327
] 

Jason Lowe commented on MAPREDUCE-6394:
---------------------------------------

Thanks for updating the patch.  I think we're almost there.

This should either call setCounters(null) instead of manipulating counters and the builder
directly or unconditionally call builder.clearCounters, otherwise we will continue to hold
onto the old counters in the builder for no good reason. 
{code}
+  public void setRawCounters(org.apache.hadoop.mapreduce.Counters rCounters) {
+    maybeInitBuilder();
+    if (counters == null) {
+      builder.clearCounters();
+    }
+    this.counters = null;
+    this.rawCounters = rCounters;
+  }
{code}

The unit tests should be using the assert methods from junit rather than native Java asserts,
as the latter implicitly relies on the JVM being run with assertions enabled for the unit
tests to function properly.  Failure to do so will have every test passing whether it should
or not.  Running with assertions enabled should happen when running under surefire and what-not,
but it's better to not have that implicit dependency.  My apologies for missing that in the
last review.

> Speed up Task processing loop in HsTasksBlock#render()
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-6394
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6394
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: jobhistoryserver
>    Affects Versions: 2.7.0
>            Reporter: Ray Chiang
>            Assignee: Ray Chiang
>              Labels: supportability
>         Attachments: MAPREDUCE-6394.001.patch, MAPREDUCE-6394.002.patch, MAPREDUCE-6394.003.patch,
MAPREDUCE-6394.004.patch, MAPREDUCE-6394.005.patch, MAPREDUCE-6394.006.patch, MAPREDUCE-6394.007.patch,
MAPREDUCE-6394.008.patch
>
>
> In HsTasksBlock#render(), there is a loop to create a Javascript table which slows down
immensely on jobs with a large number of tasks (200k or more).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message