allura-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Brondsema <d...@brondsema.net>
Subject Tests locking up with 100% CPU usage
Date Fri, 19 Sep 2014 16:18:30 GMT
Starting with Igor's comments on https://sourceforge.net/p/allura/tickets/7657/#c7d9

> There's a couple of new tests commented out in a last commit. I can't figure out why,
but they cause allura/tests/test_dispatch.py to hang when run together with other tests. Also
I have added and then removed tests for enable/disable user for the same reason.
> 
> I think it needs another pair of eyes on it, since I've already spent too much time dealing
with this tests and have no idea what's happening... Maybe I'm missing something obvious.

Alex and I have seen this recently too, and its hard to figure out what exactly
is the problem.  I first noticed it when running `./run_tests --with-coverage`
which would run nosetests in the Allura dir and would not use --processes=N
because of the with-coverage param.  So basically just a regular run of the
tests in the Allura dir would cause the CPU to go into 100% usage and the tests
wouldn't finish.  Couldn't ctrl-C or profile them, had to kill -9 it.

That was on Centos 5.10 and a workaround was to run with --processes=N and then
the tests would finish fine.  On the Ubuntu vagrant image, I didn't encounter
any problem in the first place.  So perhaps related to the environment.

I tried to narrow down to a specific test that might be the culprit.  I found
tests consistently got up to TestSecurity.test_auth (which is a bit weird and
old test anyway).  And also that commenting out that test let them all pass.

But I'm pretty sure Alex said he dug into this as well and found variation in
what tests could cause the problem.  I think he told me that going back in
git-history before the problem, and then adding a single test (a copy of an
existing one) caused the problem.  So perhaps some limit, or resource tipping
point is hit.

Alex or Igor, any more data points you know from what you've seen?

Anyone else seen anything like this?  Or have ideas for how to approach nailing
it down better?


-- 
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
              <><

Mime
View raw message