activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Async Writer Thread Shutdown error adding a message on 5.15.2 broker
Date Wed, 10 Jan 2018 07:07:06 GMT
Hmmm, I think I may have misinterpreted a few of the things you've written,
so I want to take a step back and recalibrate my understanding of the
behavior you're seeing.

On Tue, Jan 9, 2018 at 1:04 PM, neon18 <neon18@nngo.net> wrote:

> > Do you get the "thread shut down" message on the first message enqueued,
> or
> > only after a certain number of messages are sent?
> I believe, after a large number of of messages have been enqueued, is when
> it enters the OutOfMemoryError state. In another test run, I did not see
> the
> "Async Writer Thread Shutdown error", just an OutOfMemoryError.
> i.e. I see an INFO PListStore:[.../data/localhost/tmp_storage] initialized
> ... | ActiveMQ BrokerService.worker.1 message
> after about 15 seconds (with no longs in between), I see a WARN Trasnport
> Conection to: tcp://... failed: java.io.IOException: Unexpected error
> occurred: java.lang.OutOfMemoryError: Java heap space |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///...
>

So I thought I had understood that you started sending messages, and at
some point you started seeing "thread shutdown" messages in the logs, and
only after a large number of those, you finally saw an OutOfMemoryError.
That series of events seems like a straightforward problem: the thread that
would be relieving memory pressure isn't running, so it doesn't relieve
that pressure, and eventually you run out of memory. Simple,
straightforward, and the only question is why the thread isn't running when
the time comes.

But the description you just laid out 1) says that the "thread shutdown"
message doesn't necessarily occur at all under some scenarios, and 2)
doesn't clearly state the ordering of when the OOM occurs relative to the
"thread shutdown" error when they do both occur, which calls into question
my understanding that the "thread shutdown" messages were the cause of the
OOM and not its result. So first, can you please clarify the timing of the
OOM vs. the "thread shutdown" errors when they both occur? And second, we
may have to look at other possibilities for why you're running out of
memory; I'll try to poke at some of those further down in this response.


> So in this scenario, maybe it is taking longer to move the in-memory
> message
> to the temp storage as too many messages are coming in under 5.15 broker
> where as the 5.14 and 5.12 broker's might be "blocking" or throttling
> incoming messages to queue to allow for messages to be moved to temp
> storage?
>

I don't see any obvious new code paths that explicitly remove any
throttling, but I don't claim that the quick look I took this evening is
fully complete.

I do see a code path that's new in 5.15.x that would allow the thread to be
shut down if another error (either an OOM or something else) occurred while
writing messages to disk, but that code path would result in a WARN message
in the logs that starts with "Journal failed while writing at:" and your
earlier response said that you saw no other errors in the logs. So if what
you wrote earlier is accurate, I don't think this new code path could
explain what you're seeing.


> > The temp store does indeed use KahaDB as its store type. Do you see
> KahaDB
> > data files (with a .log extension) in that directory?
> Yes
> > Are they recreated if you stop the broker, delete them, and start the
> > broker? And are any bytes
> > written to them after the broker's initial startup?
> I have not tried this, should the message producer for the queue be on and
> putting stuff into the it?
>

My goal was simply to confirm that bytes were being written to the files
when messages were being enqueued, eliminating any bytes that might have
been written as part of the process of creating those files with no
messages in them, since there might be a small amount of overhead bytes in
an "empty" file. So I'd suggest you do the steps I listed without any
messages being enqueued, give the broker a few seconds (or even minutes,
why not) to let everything stabilize, then start producing messages, and
see whether bytes are eventually written to the files after that point.


> > It seems as though either the thread that dumps messages out of memory
> and
> > into the temp store isn't started at all, or is dying at some point, and
> > it
> > would be useful to know which it is. Can you attach JConsole to the
> broker
> > process as soon as possible after it starts (before sending any messages)
> > and look for the Async Writer Thread in the list of threads? If it's
> > there,
> > then try to figure out when it dies (e.g. after the first message is
> > sent).
> When I startup broker w/o any producer, there is no "Async Writer Thread",
> only an "ActiveMQ Data File Writer" thread
>

That's the one you want. When I wrote my previous response I didn't have
access to the source code, so I didn't have a way to confirm the exact name
of the thread, but what you quoted is the thread in question.


> After about 100 messages send to the non-persistent queue (the messages are
> all still in memory), there is still no "Async Writer Thread", , only an
> "ActiveMQ Data File Writer" thread and all is fine (e.g. consumers can
> de-queue messages).
> Only when I open the flood gates and push lots of messages on the
> non-persistent queue does the broker crumble with the OutOfMemoryError then
> dies in the 5.15 broker.
>

Once you do that, at what point does the "ActiveMQ Data File Writer" thread
die? Before or after you get an OOM?

Also, you say "open the flood gates." Are you trying to push an
unreasonable rate of messages at your broker? For example, are you trying
to push messages at the broker faster than (or nearly as fast as) the disk
on which your temp store is located is capable of writing them? If you send
the same number of messages without consuming them but at half the rate, or
a tenth of the rate, does the broker handle that number of messages without
OOMing?

And I say "unreasonable" not to imply that you might not have a need to
support that messaging rate, but just to imply that the computing solution
(covering both hardware and software) that you've chosen might not be
scoped for the rates you're looking for. So the rate is unreasonable for
the choices you've made (or your coworkers have made) to date, but not
necessarily that it's not one that could be met.

Finally, can you give some rough ideas of the scale you're working at? How
big of a JVM heap are you using? How big are your messages, and how many of
them are received per second? How many messages might pile up before being
consumed? What kind of storage are you writing the temp store to, and how
fast is it? Since some of the things you've written in your most recent
response sound like this might turn out to be a performance
problem/question, please provide details about the performance of the
hardware you're using as well as the scope/scale of the workload you're
trying to run.

Tim

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message