lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3667) Consider changing how we set the number of threads to use to run tests.
Date Sat, 24 Dec 2011 14:16:30 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175729#comment-13175729
] 

Mark Miller commented on LUCENE-3667:
-------------------------------------

bq. As far as the RAM usage, maybe solr tests shouldnt override lucene's default of 1 then.

Probably they should not - but I override that down to 1 in my build.properties - like I said,
that still sticks me with 8 threads. Lucene was lowered by default at some point, but I guess
no one changed Solr. I'm not too worried about what I can change without hacking the build
though. 

bq. -Dtests.sequential=true works good for me on restricted systems.

It doesn't work good for me - it takes *many* times longer than if I use 5 threads. 

Its not a 'restricted' system - like a little under powered laptop - it's a fast quad core
system that doesn't like our tests because its got 8 GB of RAM instead 16 (and I run a lot
of RAM hungry apps) and because hyper threading causes the processor count to basically be
a lie. I just want less than 8 threads because I have a sweet spot closer to 5 (I do hack
my change into the build now) - more threads are not causing faster runs and they are occasionally
bringing my comp to its knees during brief periods in the test run. I don't want one thread
though - thats torturously slow - I just want full control of the number of threads to get
below the 8 or more jail I live in.

8 threads will often bring my system to a crawl if I am trying to do something else, but 5
threads will run the tests in just about the same time for me, and it won't swamp me - but
I cannot choose 5.
                
> Consider changing how we set the number of threads to use to run tests.
> -----------------------------------------------------------------------
>
>                 Key: LUCENE-3667
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3667
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>            Assignee: Mark Miller
>            Priority: Minor
>
> The current way we set the number of threads to use is not expressive enough for some
systems. My quad core with hyper threading is recognized as 8 CPUs - since I can only override
the number of threads to use per core, 8 is as low as I can go. 8 threads can be problematic
for me - just the amount of RAM used sometimes can toss me into heavy paging because I only
have 8 GB of RAM - the heavy paging can cause my whole system to come to a crawl. Without
hacking the build, I don't think I have a lot of workarounds.
> I'd like to propose that switch from using threadsPerProcessor to threadCount. In some
ways, it's not as nice, because it does not try to scale automatically per system. But that
auto scaling is often not ideal (hyper threading, wanting to be able to do other work at the
same time), so perhaps we just default to 1 or 2 threads and devs can override individually?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message