activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From neon18 <>
Subject Re: Async Writer Thread Shutdown error adding a message on 5.15.2 broker
Date Tue, 09 Jan 2018 20:04:05 GMT
> Do you get the "thread shut down" message on the first message enqueued, or 
> only after a certain number of messages are sent? 
I believe, after a large number of of messages have been enqueued, is when
it enters the OutOfMemoryError state. In another test run, I did not see the
"Async Writer Thread Shutdown error", just an OutOfMemoryError.
i.e. I see an INFO PListStore:[.../data/localhost/tmp_storage] initialized
... | ActiveMQ BrokerService.worker.1 message
after about 15 seconds (with no longs in between), I see a WARN Trasnport
Conection to: tcp://... failed: Unexpected error
occurred: java.lang.OutOfMemoryError: Java heap space | | ActiveMQ
Transport: tcp:///...
So in this scenario, maybe it is taking longer to move the in-memory message
to the temp storage as too many messages are coming in under 5.15 broker
where as the 5.14 and 5.12 broker's might be "blocking" or throttling
incoming messages to queue to allow for messages to be moved to temp

> Have you set any options in your activemq.xml that might change the 
> broker's behavior regarding the temp store? If you can, please post your 
> configuration. 
I am pretty much using the factory stock activemq.xml, with an increased
maximumConnections and a few slow workoff queues with queuePrefetch=1

> The temp store does indeed use KahaDB as its store type. Do you see KahaDB 
> data files (with a .log extension) in that directory?
> Are they recreated if you stop the broker, delete them, and start the
> broker? And are any bytes 
> written to them after the broker's initial startup? 
I have not tried this, should the message producer for the queue be on and
putting stuff into the it?

> It seems as though either the thread that dumps messages out of memory and 
> into the temp store isn't started at all, or is dying at some point, and
> it 
> would be useful to know which it is. Can you attach JConsole to the broker 
> process as soon as possible after it starts (before sending any messages) 
> and look for the Async Writer Thread in the list of threads? If it's
> there, 
> then try to figure out when it dies (e.g. after the first message is
> sent). 
When I startup broker w/o any producer, there is no "Async Writer Thread",
only an "ActiveMQ Data File Writer" thread
After about 100 messages send to the non-persistent queue (the messages are
all still in memory), there is still no "Async Writer Thread", , only an
"ActiveMQ Data File Writer" thread and all is fine (e.g. consumers can
de-queue messages).
Only when I open the flood gates and push lots of messages on the
non-persistent queue does the broker crumble with the OutOfMemoryError then
dies in the 5.15 broker.

Thanks for looking into this issue with me.



Sent from:

View raw message