hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
Date Wed, 05 Aug 2009 18:07:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739650#action_12739650

Aaron Kimball commented on MAPREDUCE-825:

Agreed about the other timeout. I'll include that as well.

I can change the suggested test, but as you say, it'll have to be a manual verification and
that'll itself be subjective. Thread.sleep(n) guarantees that a thread will sleep for at least
'n' milliseconds, or receive an InterruptedException. But the OS is always free to reschedule
your thread later, and there are a lot of other variable-time components in these tests. So
writing JUnit tests that compare relative runtimes might nondeterministically fail.

I'll report back results.

> JobClient completion poll interval of 5s causes slow tests in local mode
> ------------------------------------------------------------------------
>                 Key: MAPREDUCE-825
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-825
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Minor
>         Attachments: completion-poll-interval.patch
> The JobClient.NetworkedJob.waitForCompletion() method polls for job completion every
5 seconds. When running a set of short tests in pseudo-distributed mode, this is unnecessarily
slow and causes lots of wasted time. When bandwidth is not scarce, setting the poll interval
to 100 ms results in a 4x speedup in some tests.  This interval should be parametrized to
allow users to control the interval for testing purposes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message