activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: ActiveMQ persistence store is full. Stopping producer
Date Tue, 02 Jul 2019 13:24:03 GMT
I'm confused. You say that a few hundred messages are processed, but the
stats you quoted show 10 million messages dequeued (processed). Was that
due to messages consumed before the work you describe began?

My guess is that at this point you can't make progress because your
transformation consumer can't write to queue 2 because the data store is
full.

You have a few options:
1. Rewrite the consumers to put both transformation and the database write
into the same process without writing to queue 2. This would allow you to
resume processing without changing any of the broker's KahaDB settings.
2. Increase the KahaDB limit on the entire store to make it big enough to
hold all messages that might be unconsumed at any time. This requires you
be able to estimate the number of messages that might be unconsumed on all
queues and the size of those messages, which might or might not be
possible, and it requires that you have that much available disk space. If
you can do this, then your producer can produce as fast as it can and your
consumers can consume when they get around to it, but if you get it wrong
then you'll end up stuck like you currently are.
3. Set a per-destination limit for queue 1 that is lower than the
entire-store limit, so that it will always be possible to write to queue 2
as long as the consumer for queue 2 is consuming messages. To make this
work from where you are now, you'll need to increase the whole-store limit
at least a little (a few GB, maybe) at the same time. This option would
allow you to process these messages without significantly increasing disk
usage, but it doesn't allow the producers to finish their work until the
consumers have consumed a significant portion of the messages, and it's
going to be painful to maintain if those additional applications you
mentioned are each going to have their own queue 1 and queue 2.

Overall, I think you should spend the time estimating your storage needs
for ActiveMQ based on how many messages will be unconsumed at any time,
particularly since you say that you'll have more of these processing
threads in the future. It sounds to me like maybe this is the first time
that you're considering that particular non-functional system requirement,
and I think you should do that before deciding on a path forward, because
what you realize from that evaluation will influence how you value the
three options I listed.

Tim

On Tue, Jul 2, 2019, 1:21 AM Kushal Gautam <kushal.gautam@gmail.com> wrote:

> Hi again:
>
> Thank you for your reply.
>
> By "stuck" I meant literally stuck. No messages were being processed.
>
> For instance, below is the queue stats of Queue#1
>
> Pending            No. of Consumers  Enqueued            Dequeued
> 8016495          1                          18094028           10077534
>
> The de-queuing process was stuck. So, was the de-queuing process of Queue#2
>
> Karaf logs did not show any signs of issues. Thus, after looking at broker
> logs, I found the following log message (as I shared earlier):
>
> > Usage(default:store:queue://tst.tst.users:store) percentUsage=99%,
> > usage=42792917913, limit=40692963880,
> > percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%,
> > usage=42792917913, limit=40692963880, percentUsageMinDelta=1%: Persistent
> > store is Full, 100% of 40692963880. Stopping producer
> > (ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding
> > queue://tst.tst.users. See
> > http://activemq.apache.org/producer-flow-control.html for more info
> > (blocking for: 228222s) | org.apache.activemq.broker.region.Queue |
> > ActiveMQ
> > Transport: tcp:///192.168.7.98:51461@61616
>
> Right now, around 200-300 messages get de-queued from Queue#1, while just
> about 100-150 messages get de-queued from Queue#2. That's why I assumed
> that the bottleneck could have been the database endpoint.
>
> The number of queues in this broker will grow significantly in upcoming
> days, with the increasing number of my karaf instances. But, the issue that
> I experienced about producer-flow-control is specific to the limit of a
> queue but not with the number of queues, right?
>
> So far, in the broker optimization process, I have only enabled
> optimizedDispatch. I am yet to measure the performance benefits of usage of
> NIO.
>
> Regards,
> Cooshal.
>
> On Tue, Jul 2, 2019 at 3:13 AM Tim Bain <tbain@alumni.duke.edu> wrote:
>
> > I had assumed that messages were being processed, just not as fast as you
> > wanted (and the producer got limited to the rate at which the consumer
> > processed them), but you said the word "stuck" several times. Which is
> it?
> > Are you not processing messages, or are you not processing messages
> > quickly? I want to make sure we understand the behavior you're actually
> > observing, since that will influence any advice we may be able to give
> you.
> >
> > You asked how to change the producer flow control limit for the
> persistent
> > store. The configuration snippet you referenced is the correct way to set
> > that.
> >
> > Your description of your processing flow and the fact that your backup
> > occurs on queue #1 means that the write to the database is not your
> > performance limiter, because if it was you'd see a backup on queue #2.
> > Rather, your consumer that translates messages from queue #1 and writes
> > them to queue #2 seems like it's not keeping up, because if it was then
> > messages written to queue #1 wouldn't back up there (instead they'd back
> up
> > on queue #2).
> >
> > Tim
> >
> > On Mon, Jul 1, 2019, 8:25 AM <kushal.gautam@gmail.com> wrote:
> >
> >> Hi:
> >>
> >> thank you for your reply.
> >>
> >> My messages to the forum are being bounced back. Thus, I am sending this
> >> directly to you.
> >>
> >> Option 1 (do nothing and wait) did nothing, as all my messages were
> stuck
> >> in-flight, and I realized that this was stuck for more than 7 hours.
> >> Restarting camel contexts or instances did not help either.
> >>
> >> Yes, there is a big gap between the rates of consumption and production.
> >> My assumption is that the bottleneck due to the database endpoint. I am
> not
> >> able to profile this behaviour.
> >>
> >> My current process is:
> >>
> >> DB Consumer (from database) -> queue 1
> >> Queue 1 consumer (consume from queue 1) -> transform the data -> queue
2
> >> Queue 2 consumer (consume from queue 2) -> write to DB
> >>
> >> and this has resulted in millions of records to be stuck in queue 1 as
> >> pending.
> >>
> >> I have restarted the broker, and it started consuming these messages
> now.
> >>
> >> How do I increase the limit for persistent store?
> >>
> >> <storeUsage>
> >>     <storeUsage limit="100 gb"/>
> >> </storeUsage>
> >>
> >> Is this the right config to change?
> >>
> >> Regards,
> >> Cooshal.
> >>
> >> <quote author='Tim Bain'>
> >> It appears that your consumer isn't keeping up with your producer(s),
> >> filling the persistent store, which slows your producer(s) to the rate
> at
> >> which the consumer is consuming.
> >>
> >> Your options are:
> >> * Do nothing, and let it complete at the current rate.
> >> * Speed up consumption, either by adding consumers or making them faster
> >> or
> >> both.
> >> * Increase the limit for the persistent store to allow the producer to
> >> write all remaining messages to the broker, so that the producer process
> >> can exit even though the consumer is still processing messages. Note
> that
> >> this requires a broker restart, so make sure your producer logic will
> >> gracefully handle a period of broker unavailability. Persistent messages
> >> will remain on the broker when a restart occurs, and since it's the
> >> persistent store that is full, all of those messages will be kept, but
> if
> >> you have any unconsumed non-persistent messages, they'll be lost.
> >>
> >> Tim
> >>
> >> On Mon, Jul 1, 2019, 5:51 AM Kushal Gautam <kushal.gautam@gmail.com>
> >> wrote:
> >>
> >> > Hi:
> >> >
> >> > I have similar issues (
> >> >
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/Persistent-store-is-Full-100-of-107374182400-Stopping-producer-td4740024.html
> >> > ).
> >> > My activemq broker was mediating the transfer of millions of records,
> >> > which
> >> > I was importing via Apache Camel (running on a
> >> > Karaf instance).
> >> >
> >> > I am currently using ActiveMQ 5.15.4, and below is how my systemUsage
> >> > config
> >> > looks like:
> >> >
> >> > <systemUsage>
> >> >     <systemUsage>
> >> >         <memoryUsage>
> >> >             <memoryUsage percentOfJvmHeap="70" />
> >> >         </memoryUsage>
> >> >         <storeUsage>
> >> >             <storeUsage limit="100 gb"/>
> >> >         </storeUsage>
> >> >         <tempUsage>
> >> >             <tempUsage limit="50 gb"/>
> >> >         </tempUsage>
> >> >     </systemUsage>
> >> > </systemUsage>
> >> >
> >> > This is what I got in the logs:
> >> >
> >> > Usage(default:store:queue://tst.tst.users:store) percentUsage=99%,
> >> > usage=42792917913, limit=40692963880,
> >> > percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%,
> >> > usage=42792917913, limit=40692963880, percentUsageMinDelta=1%:
> >> Persistent
> >> > store is Full, 100% of 40692963880. Stopping producer
> >> > (ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding
> >> > queue://tst.tst.users. See
> >> > http://activemq.apache.org/producer-flow-control.html for more info
> >> > (blocking for: 228222s) | org.apache.activemq.broker.region.Queue |
> >> > ActiveMQ
> >> > Transport: tcp:///192.168.7.98:51461@61616
> >> >
> >> > I have restarted the karaf instances, but this does not seem to take
> an
> >> > effect, and after looking at the broker logs, it quite made sense.
> >> >
> >> > What can I actually do to avert this problem?
> >> >
> >> > Currently, the broker stats for my queue shows:
> >> > Pending            No. of Consumers  Enqueued            Dequeued
> >> > 8016495          1                          18094028
>  10077534
> >> >
> >> > I am not sure, if restarting the broker will result in loss of these
> >> > messages.
> >> >
> >> > Also, would be great if you could suggest about the config options
> that
> >> I
> >> > can make to optimize it better.
> >> >
> >> > Regards,
> >> > Cooshal.
> >> >
> >>
> >> </quote>
> >> Quoted from:
> >>
> >>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-persistence-store-is-full-Stopping-producer-tp4751371p4751374.html
> >>
> >>
> >> _____________________________________
> >> Sent from http://activemq.2283324.n4.nabble.com
> >>
> >>
> > On Mon, Jul 1, 2019, 8:25 AM <kushal.gautam@gmail.com> wrote:
> >
> >> Hi:
> >>
> >> thank you for your reply.
> >>
> >> My messages to the forum are being bounced back. Thus, I am sending this
> >> directly to you.
> >>
> >> Option 1 (do nothing and wait) did nothing, as all my messages were
> stuck
> >> in-flight, and I realized that this was stuck for more than 7 hours.
> >> Restarting camel contexts or instances did not help either.
> >>
> >> Yes, there is a big gap between the rates of consumption and production.
> >> My assumption is that the bottleneck due to the database endpoint. I am
> not
> >> able to profile this behaviour.
> >>
> >> My current process is:
> >>
> >> DB Consumer (from database) -> queue 1
> >> Queue 1 consumer (consume from queue 1) -> transform the data -> queue
2
> >> Queue 2 consumer (consume from queue 2) -> write to DB
> >>
> >> and this has resulted in millions of records to be stuck in queue 1 as
> >> pending.
> >>
> >> I have restarted the broker, and it started consuming these messages
> now.
> >>
> >> How do I increase the limit for persistent store?
> >>
> >> <storeUsage>
> >>     <storeUsage limit="100 gb"/>
> >> </storeUsage>
> >>
> >> Is this the right config to change?
> >>
> >> Regards,
> >> Cooshal.
> >>
> >> <quote author='Tim Bain'>
> >> It appears that your consumer isn't keeping up with your producer(s),
> >> filling the persistent store, which slows your producer(s) to the rate
> at
> >> which the consumer is consuming.
> >>
> >> Your options are:
> >> * Do nothing, and let it complete at the current rate.
> >> * Speed up consumption, either by adding consumers or making them faster
> >> or
> >> both.
> >> * Increase the limit for the persistent store to allow the producer to
> >> write all remaining messages to the broker, so that the producer process
> >> can exit even though the consumer is still processing messages. Note
> that
> >> this requires a broker restart, so make sure your producer logic will
> >> gracefully handle a period of broker unavailability. Persistent messages
> >> will remain on the broker when a restart occurs, and since it's the
> >> persistent store that is full, all of those messages will be kept, but
> if
> >> you have any unconsumed non-persistent messages, they'll be lost.
> >>
> >> Tim
> >>
> >> On Mon, Jul 1, 2019, 5:51 AM Kushal Gautam <kushal.gautam@gmail.com>
> >> wrote:
> >>
> >> > Hi:
> >> >
> >> > I have similar issues (
> >> >
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/Persistent-store-is-Full-100-of-107374182400-Stopping-producer-td4740024.html
> >> > ).
> >> > My activemq broker was mediating the transfer of millions of records,
> >> > which
> >> > I was importing via Apache Camel (running on a
> >> > Karaf instance).
> >> >
> >> > I am currently using ActiveMQ 5.15.4, and below is how my systemUsage
> >> > config
> >> > looks like:
> >> >
> >> > <systemUsage>
> >> >     <systemUsage>
> >> >         <memoryUsage>
> >> >             <memoryUsage percentOfJvmHeap="70" />
> >> >         </memoryUsage>
> >> >         <storeUsage>
> >> >             <storeUsage limit="100 gb"/>
> >> >         </storeUsage>
> >> >         <tempUsage>
> >> >             <tempUsage limit="50 gb"/>
> >> >         </tempUsage>
> >> >     </systemUsage>
> >> > </systemUsage>
> >> >
> >> > This is what I got in the logs:
> >> >
> >> > Usage(default:store:queue://tst.tst.users:store) percentUsage=99%,
> >> > usage=42792917913, limit=40692963880,
> >> > percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%,
> >> > usage=42792917913, limit=40692963880, percentUsageMinDelta=1%:
> >> Persistent
> >> > store is Full, 100% of 40692963880. Stopping producer
> >> > (ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding
> >> > queue://tst.tst.users. See
> >> > http://activemq.apache.org/producer-flow-control.html for more info
> >> > (blocking for: 228222s) | org.apache.activemq.broker.region.Queue |
> >> > ActiveMQ
> >> > Transport: tcp:///192.168.7.98:51461@61616
> >> >
> >> > I have restarted the karaf instances, but this does not seem to take
> an
> >> > effect, and after looking at the broker logs, it quite made sense.
> >> >
> >> > What can I actually do to avert this problem?
> >> >
> >> > Currently, the broker stats for my queue shows:
> >> > Pending            No. of Consumers  Enqueued            Dequeued
> >> > 8016495          1                          18094028
>  10077534
> >> >
> >> > I am not sure, if restarting the broker will result in loss of these
> >> > messages.
> >> >
> >> > Also, would be great if you could suggest about the config options
> that
> >> I
> >> > can make to optimize it better.
> >> >
> >> > Regards,
> >> > Cooshal.
> >> >
> >>
> >> </quote>
> >> Quoted from:
> >>
> >>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-persistence-store-is-full-Stopping-producer-tp4751371p4751374.html
> >>
> >>
> >> _____________________________________
> >> Sent from http://activemq.2283324.n4.nabble.com
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message