incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-42) Improve speed of running integration tests
Date Tue, 14 Aug 2012 18:37:38 GMT


Gabriel Reid commented on CRUNCH-42:

Yep, it seems to be due to a change in the dependencies -- running with hadoop 1.0.3 still
seems to be pretty slow, and running with CDH3 (the version that was in the pom when I made
the original patch) was fast. Using hadoop 2.0 also seems to run quickly with this change.
I was actually starting to think that I had imagined running "mvn verify" in under 10 minutes,
and it was starting to worry me :-)

In any case, I believe that there's a config setting that can be set somewhere in hadoop 1.x
to change the poll interval as well -- I'm currently trying to hunt that down and find the
best place to set it, and then the performance improvement should work everywhere.
> Improve speed of running integration tests
> ------------------------------------------
>                 Key: CRUNCH-42
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>    Affects Versions: 0.3.0
>            Reporter: Gabriel Reid
>            Assignee: Gabriel Reid
>            Priority: Minor
>         Attachments: 0001-Use-short-job-poll-interval-when-in-local-mode.patch
> The CrunchJobControl currently uses a hard-coded poll interval of 5 seconds to check
on running jobs. This is valid when running jobs on large datasets on a cluster, but results
in unnecessary waiting when running in local mode. By using a smaller poll interval when running
in local mode, the performance of the integration tests can be greatly improved.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message