Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Date: Wed, 15 Aug 2012 07:29:38 +1100 (NCT)
From: "Gabriel Reid (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <1239153357.9569.1344976178145.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (CRUNCH-42) Improve speed of running integration
 tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434477#comment-13434477 ] 

Gabriel Reid commented on CRUNCH-42:
------------------------------------

It seems that there is still a hardcoded sleep and interval in Hadoop 1.0.3 that isn't present in 2.0 or in CDH3, so the performance improvements will just remain much larger when running with either 2.0 or CDH3. I've pushed this to the main repo as it seems that there isn't anything that can be done for 1.0.3 (and it still provides a verify small improvement). 

Running with 2.0 or CDH3 allows running the crunch integration tests in something under 7 minutes on my laptop.
                
> Improve speed of running integration tests
> ------------------------------------------
>
>                 Key: CRUNCH-42
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-42
>             Project: Crunch
>          Issue Type: Improvement
>    Affects Versions: 0.3.0
>            Reporter: Gabriel Reid
>            Assignee: Gabriel Reid
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: 0001-Use-short-job-poll-interval-when-in-local-mode.patch
>
>
> The CrunchJobControl currently uses a hard-coded poll interval of 5 seconds to check on running jobs. This is valid when running jobs on large datasets on a cluster, but results in unnecessary waiting when running in local mode. By using a smaller poll interval when running in local mode, the performance of the integration tests can be greatly improved.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira