mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAHOUT-916) Make Mahout's tests run in parallel
Date Wed, 05 Jun 2013 07:45:21 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675671#comment-13675671
] 

Sean Owen commented on MAHOUT-916:
----------------------------------

After looking at this again I am pretty certain the ultimate cause is memory. There is a load
of swapping going on for me, and while the jobs are generating a lot of I/O, it's swapping
that is probably overwhelming the disk. An SSD would certainly do better but it isn't the
issue. 

On a 4-core machine this is going to try to run 6 additional JVMs with max heap size of 1.8GB.
Figuring 25% Java overhead an leaving a GB for everything else on the system, this needs 14.5GB
to run comfortably. That seems pretty high as a default requirement.

But I appear to be running all these successfully with a max heap size of 768M. That works
comfortably with 8GB RAM, no meaningful swapping. How about making that revision to bring
the default requirement down?

In fact, if you change this to use 1 JVM per core (vs 1.5) it would also squeeze onto a 4GB
/ 4-core machine.
                
> Make Mahout's tests run in parallel
> -----------------------------------
>
>                 Key: MAHOUT-916
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-916
>             Project: Mahout
>          Issue Type: Improvement
>          Components: build
>            Reporter: Grant Ingersoll
>            Assignee: Isabel Drost-Fromm
>            Priority: Minor
>              Labels: MAHOUT_INTRO_CONTRIBUTE
>             Fix For: 0.8
>
>         Attachments: MAHOUT-916.patch, MAHOUT-916.patch, MAHOUT-916.patch, MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests.  We should hook this in to Mahout.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message