qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Jones" <jimmyjon...@gmx.co.uk>
Subject Re: System stalling
Date Wed, 21 Aug 2013 16:08:02 GMT
>>>> I've got an simple processing system using the 0.22 C++ broker, all
>>>> on one box, where an external system posts messages to the default
>>>> headers exchange, and an ingest process receives them using a ring
>>>> queue, transforms them and outputs to a different headers exchange.
>>>> Various other processes pick messages of interest off that exchange
>>>> using ring queues. Recently however the system has been stalling -
>>>> I'm still receiving lots of data from the other system, but the
>>>> ingest process suddenly goes to <5% CPU usage and its queue fills up
>>>> and messages start getting discarded from the ring, the follow on
>>>> processes go to practically 0% CPU and qpidd hovers around 95-120%
>>>> CPU (normally its ~75%) and the rest of the system pretty much goes
>>>> idle (no swapping, there is free memory)
>>>>
>>>> I attached to the ingest process with gdb and it was stuck in send
>>>> (waitForCapacity/waitForCompletionImpl) - I notice this can block.
>>>
>>> Is there any queue bound to the second headers exchange, i.e. to the one
>>> this ingest process is sending to, that is not a ring queue? (If you run
>>> qpid-config queue -r, you get a quick listing of the queues and their
>>> bindings).
>>
>> I've run qpid-config queue, and all my queues have --limit-policy=ring, apart
>> from a UUID one which I presume is qpid-config itself. Are there any other useful
>> debugging things I can do?
>
>What does qpid-stat -q show? Is it possible to test whether the broker 
>is still responsive e,g, by sending and receiving messages through a 
>test queue/exchange? Are there any errors in the logs? Are any of the 
>queues durable (and messages persistent)?

qpid-stat -q is all zero's in the msg & bytes column, apart from the ingest queue,
and another overflowing ring queue I have.

I did run qpid-tool when the system was broken to dump some stats. msgTotalDequeues
was slowly incremeneting on the ingest queue, so I presume messages were still being
delivered and the broker was responsive?

The only logging I've got is syslog, and I just see a warning about unsent data,
presumably when the ingest process receives a SIGALARM. I'm happy to swich on more
logging, what would you recommend?

None of my queues are durable, but I think incoming messages from the other system
are marked as durable. The exchange that the ingest process sends to is durable,
but I'm not setting any durable flags on outgoing messages (I presume the default
is off).

>Another thing might be a ptrace of the broker process. Maybe two or 
>three with a short delay between them.

I'll try this next time it goes haywire.

>For some reason it seems like the broker is not sending back 
>confirmation to the sender in the ingest process, causing that to block. 
>Ring queues shouldn't be subject to producer flow control so we need to 
>figure out what other reason there could be for that.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message