hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
Date Thu, 06 Aug 2009 19:17:14 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Aaron Kimball updated MAPREDUCE-825:

    Attachment: MAPREDUCE-825.2.patch

Attaching a new patch that also includes a new "jobclient.progress.monitor.poll.interval"
setting; default is 1000 ms. Modified TestTaskFail to set the completion poll interval to
50 ms.

With default (5000) ms timeout, test runtime was 3 minutes 15 seconds. Setting the timeout
to 50 ms reduced test runtime to 3 minutes 8 seconds. If we expect an average of 2500 milliseconds
wasted per job in the default case, then this is 2500*3 = 7500 ms expected to be wasted, so
the observed speedup seems correct. To be sure, I also set the timeout to 20000 ms; test runtime
went up to 3 minutes 52 seconds. So there's definitely a correlation.

> JobClient completion poll interval of 5s causes slow tests in local mode
> ------------------------------------------------------------------------
>                 Key: MAPREDUCE-825
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-825
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Minor
>         Attachments: completion-poll-interval.patch, MAPREDUCE-825.2.patch
> The JobClient.NetworkedJob.waitForCompletion() method polls for job completion every
5 seconds. When running a set of short tests in pseudo-distributed mode, this is unnecessarily
slow and causes lots of wasted time. When bandwidth is not scarce, setting the poll interval
to 100 ms results in a 4x speedup in some tests.  This interval should be parametrized to
allow users to control the interval for testing purposes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message