qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Jones" <jimmyjon...@gmx.co.uk>
Subject System stalling
Date Wed, 21 Aug 2013 12:39:06 GMT

I've got an simple processing system using the 0.22 C++ broker, all on one box, where an external
system posts messages to the default headers exchange, and an ingest process receives them
using a ring queue, transforms them and outputs to a different headers exchange. Various other
processes pick messages of interest off that exchange using ring queues. Recently however
the system has been stalling - I'm still receiving lots of data from the other system, but
the ingest process suddenly goes to <5% CPU usage and its queue fills up and messages start
getting discarded from the ring, the follow on processes go to practically 0% CPU and qpidd
hovers around 95-120% CPU (normally its ~75%) and the rest of the system pretty much goes
idle (no swapping, there is free memory)

I attached to the ingest process with gdb and it was stuck in send (waitForCapacity/waitForCompletionImpl)
- I notice this can block. However given the rest of the system is idle when this problem
occurs I can't understand why this would happen. I added a SIGALARM handler around send with
a timeout of 30s and the process did sometimes get killed. Looking at qpid-tool it does seem
to still be processing messages, just extremely slowly. My other observation is from netstat,
the Send-Q of qpidd to the ingest process is 16363, and the Recv-Q and Send-Q of the ingest
process are both 0.

Any ideas on what might be happening are very welcome!



To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

View raw message