allura-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Brondsema <>
Subject Re: Tests locking up with 100% CPU usage
Date Fri, 19 Sep 2014 21:25:10 GMT
On 9/19/14 12:18 PM, Dave Brondsema wrote:
> Starting with Igor's comments on
>> There's a couple of new tests commented out in a last commit. I can't figure out
why, but they cause allura/tests/ to hang when run together with other tests.
Also I have added and then removed tests for enable/disable user for the same reason.
>> I think it needs another pair of eyes on it, since I've already spent too much time
dealing with this tests and have no idea what's happening... Maybe I'm missing something obvious.
> Alex and I have seen this recently too, and its hard to figure out what exactly
> is the problem.  I first noticed it when running `./run_tests --with-coverage`
> which would run nosetests in the Allura dir and would not use --processes=N
> because of the with-coverage param.  So basically just a regular run of the
> tests in the Allura dir would cause the CPU to go into 100% usage and the tests
> wouldn't finish.  Couldn't ctrl-C or profile them, had to kill -9 it.
> That was on Centos 5.10 and a workaround was to run with --processes=N and then
> the tests would finish fine.  On the Ubuntu vagrant image, I didn't encounter
> any problem in the first place.  So perhaps related to the environment.
> I tried to narrow down to a specific test that might be the culprit.  I found
> tests consistently got up to TestSecurity.test_auth (which is a bit weird and
> old test anyway).  And also that commenting out that test let them all pass.
> But I'm pretty sure Alex said he dug into this as well and found variation in
> what tests could cause the problem.  I think he told me that going back in
> git-history before the problem, and then adding a single test (a copy of an
> existing one) caused the problem.  So perhaps some limit, or resource tipping
> point is hit.
> Alex or Igor, any more data points you know from what you've seen?
> Anyone else seen anything like this?  Or have ideas for how to approach nailing
> it down better?

I tried checking out branch je/42cc_7657 and going back to commit
4cc63586e5728d7d0c5c2f09150eb07eb7e4edc1 (before tests were commented out) to
see what happened for me:

On vagrant / ubuntu, it froze at same as you.  So some
consistency there.  Tests passed when I ran `nosetests --process-timeout=180
--processes=4 -v` in the Allura dir.  Seemed slow at the end though, almost
thought it froze.

On centos, it froze at a different spot with a regular nosetests run.  It passed
with `nosetests allura/tests/ --processes=4 --process-timeout=180 -v`.  For some
reason (hopefully unrelated), I needed to specify path "allura/tests/" to avoid
an IOError from multiprocessing.

So at least multiprocess tests still seems like a workaround for me.  Note:
./run_tests picks a --processes=N value dynamically based on the machine's CPU
cores, so with a single core you don't get multiple processes that way.  Also
note: if you have nose-progressive installed and active, that is incompatible
with multiple processes.

Dave Brondsema : : personal : programming

View raw message