incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-42) Improve speed of running integration tests
Date Tue, 14 Aug 2012 20:29:38 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434477#comment-13434477
] 

Gabriel Reid commented on CRUNCH-42:
------------------------------------

It seems that there is still a hardcoded sleep and interval in Hadoop 1.0.3 that isn't present
in 2.0 or in CDH3, so the performance improvements will just remain much larger when running
with either 2.0 or CDH3. I've pushed this to the main repo as it seems that there isn't anything
that can be done for 1.0.3 (and it still provides a verify small improvement). 

Running with 2.0 or CDH3 allows running the crunch integration tests in something under 7
minutes on my laptop.
                
> Improve speed of running integration tests
> ------------------------------------------
>
>                 Key: CRUNCH-42
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-42
>             Project: Crunch
>          Issue Type: Improvement
>    Affects Versions: 0.3.0
>            Reporter: Gabriel Reid
>            Assignee: Gabriel Reid
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: 0001-Use-short-job-poll-interval-when-in-local-mode.patch
>
>
> The CrunchJobControl currently uses a hard-coded poll interval of 5 seconds to check
on running jobs. This is valid when running jobs on large datasets on a cluster, but results
in unnecessary waiting when running in local mode. By using a smaller poll interval when running
in local mode, the performance of the integration tests can be greatly improved.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message