activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Understanding memoryLimits with flow control and systemUsage limits.
Date Wed, 05 Jun 2013 15:57:01 GMT
Sounds like it's happening exactly as you have it configured.. I assume
you're using non-persistent messages?

As Johan pointed out, your config kinda doesn't make sense :)

The <memoryUsage> is an overall broker limit. That is, if you set your
<memoryUsage limit="40mb"> then you want to keep the aggregate of all of
your destinations <= 40mb. Therefore, if you have 40 queues, set each one
to 1mb. Or some other combo that is <= 40mb. In your case you set each
queue to 50mb (<policyEntry memoryLimit="50mb"...) which doesn't work well.
As for your questions...

1) the 50mb limit doesn't kick in (you don't get PFC) because the broker
has hit the 70% limit and is spooling your messages to disk. in the first
case, it never hits the 70% mem limit of your destination because it filled
up (40mb) the broker main memory first

2) it sure is happening :) check the logs for scenario #2... you hit the
limit on the temp store

3) that doesn't make sense. whip up a test case to show that.

hope that helps :)


On Wed, Jun 5, 2013 at 8:39 AM, Johan Edstrom <seijoed@gmail.com> wrote:

> Do it the other way around.
> The system and store settings are global, the others per destination.
> So what you are doing doesn't make a ton of sense.
>
>
> On Jun 5, 2013, at 9:24 AM, Jesus Roncero <jesus@we7.com> wrote:
>
> > Hi all,
> >
> > I've got a testing activeMQ installation that I'm trying to test
> regarding
> > different limits on destinations and memory usage in general, with and
> without
> > flowControl enable. I'm running into some issues and I don't quite
> understand
> > how this works, so I would appreciate any tips or advise here.
> >
> > My setup is activeMQ 5.6 in a testing environment. I'm using a couple of
> > python scripts that use stomp.py to send some messages and read them
> from a
> > testqueue. Depending on a couple of settings, the behaviour changes, and
> it
> > behaves differently.
> >
> > I'm sendind 1 Mb messages to the queue as per:
> >
> > data = "x" * 1024 * 1024
> > number = 1000
> > print "producing %s %s kb messages" % (number, len(data) / 1024)
> > for i in range(number):
> >    print i
> >    conn.send(data, destination='/queue/testqueue')
> >
> > in python.
> >
> > Now, I have two settings:
> >
> > Scenario number 1
> > ----------------
> >
> > <policyEntries>
> >     <policyEntry queue=">" memoryLimit="50mb" producerFlowControl="true">
> >         <pendingQueuePolicy>
> >             <fileQueueCursor />
> >         </pendingQueuePolicy>
> >     </policyEntry>
> >     <policyEntry topic=">" memoryLimit="50mb" producerFlowControl="true">
> >         <!-- We want to store both persistent and non-persistent
> messages -->
> >         <pendingSubscriberPolicy>
> >             <fileCursor />
> >         </pendingSubscriberPolicy>
> >         <pendingDurableSubscriberPolicy>
> >             <fileDurableSubscriberCursor/>
> >         </pendingDurableSubscriberPolicy>
> >
> >         ...
> >
> >        <systemUsage>
> >            <systemUsage>
> >                <memoryUsage>
> >                    <memoryUsage limit="40 mb"/>
> >                </memoryUsage>
> >                <storeUsage>
> >                    <storeUsage limit="1 gb" />
> >                </storeUsage>
> >                <tempUsage>
> >                    <tempUsage limit="800 mb"/>
> >                </tempUsage>
> >            </systemUsage>
> >        </systemUsage>
> >
> > Where the memoryLimit of each of the queues is bigger than the limit for
> > memoryUsage. When I try to insert into the queue, it insert 32 messages
> (32Mb)
> > and it hangs in there. I see this log messages:
> >
> > [2013-06-05 15:54:19,816] INFO ActiveMQ Transport: tcp:///
> 10.0.9.124:36431 org.apache.activemq.broker.region.Queue - Usage Manager
> Memory Limit (52428800) reached on queue://testqueue. Producers will be
> throttled to the rate at which messages are removed from this destination
> to prevent flooding it. See
> http://activemq.apache.org/producer-flow-control.html for more info
> > 2013-06-05 15:54:19,816 [0.0.9.124:36431] INFO  Queue  - Usage Manager
> Memory Limit (52428800) reached on queue://testqueue. Producers will be
> throttled to the rate at which messages are removed from this destination
> to prevent flooding it. See
> http://activemq.apache.org/producer-flow-control.html for more info
> > [2013-06-05 15:54:20,818] INFO ActiveMQ Transport: tcp:///
> 10.0.9.124:36431 org.apache.activemq.broker.region.Queue -
> Usage(default:memory:queue://testqueue:memory) percentUsage=62%,
> usage=32537724, limit=52428800,
> percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%,
> usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager
> Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1)
> stopped to prevent flooding queue://testqueue. See
> http://activemq.apache.org/producer-flow-control.html for more info
> (blocking for: 1s)
> > 2013-06-05 15:54:20,818 [0.0.9.124:36431] INFO  Queue  -
> Usage(default:memory:queue://testqueue:memory) percentUsage=62%,
> usage=32537724, limit=52428800,
> percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%,
> usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager
> Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1)
> stopped to prevent flooding queue://testqueue. See
> http://activemq.apache.org/producer-flow-control.html for more info
> (blocking for: 1s)
> >
> > Which would indicate that flow control kicked in as per the first
> message,
> > that they reach a memory limit (50Mb) as per the per destination limit
> (second
> > message). It also says that the memory (40Mb) got used as a 100%.
> >
> > Also, noted that it says: default:memory:queue.
> >
> >
> > Scenario number 2
> > -----------------
> >
> > Now, if I change the memoryUsage to be slightly bigger than the
> memoryLimit on
> > each queue, from 40 to 60Mb, as per:
> >
> >        <systemUsage>
> >            <systemUsage>
> >                <memoryUsage>
> >                    <memoryUsage limit="60 mb"/>
> >                </memoryUsage>
> >                <storeUsage>
> >                    <storeUsage limit="1 gb" />
> >                </storeUsage>
> >                <tempUsage>
> >                    <tempUsage limit="800 mb"/>
> >                </tempUsage>
> >            </systemUsage>
> >        </systemUsage>
> >
> > taking into account that the per-destination limits are just the same,
> 50Mb as
> > per the config above. When I run the same exact procedure, it inserts
> 801 1Mb
> > messages and it stops with this message:
> >
> > [2013-06-05 16:05:01,972] INFO BrokerService.worker.1
> org.apache.activemq.store.kahadb.plist.PListStore -
> PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
> > 2013-06-05 16:05:01,972 [ervice.worker.1] INFO  PListStore -
> PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
> > [2013-06-05 16:05:12,959] INFO ActiveMQ Transport: tcp:///
> 10.0.9.124:36615 org.apache.activemq.broker.region.Queue -
> Usage(default:temp:queue://testqueue:temp) percentUsage=99%,
> usage=839122944, limit=838860800,
> percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%,
> usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is
> Full (99% of 838860800). Stopping producer
> (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding
> queue://testqueue. See
> http://activemq.apache.org/producer-flow-control.html for more info
> (blocking for: 1s)
> > 2013-06-05 16:05:12,959 [0.0.9.124:36615] INFO  Queue -
> Usage(default:temp:queue://testqueue:temp) percentUsage=99%,
> usage=839122944, limit=838860800,
> percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%,
> usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is
> Full (99% of 838860800). Stopping producer
> (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding
> queue://testqueue. See
> http://activemq.apache.org/producer-flow-control.html for more info
> (blocking for: 1s)
> >
> > Where I see that it starts using the tempUsage space, and it pauses when
> it's
> > used the 800 Mb available, whining that the temp store is full.
> >
> > Now, the type is default:temp:queue rather than memory
> >
> > So,
> >
> > Could anyone explain to me why this is? I've read quite a few blog posts
> and
> > entries on the mailing list and I'm still a bit confused about it, so,
> to have
> > more concrete questions,
> >
> > 1) Why this change on the memoryUsage limit affects how activeMQ deals
> with
> > way too many messages? More so, why on the second example, the 50 Mb
> limit
> > doesn't kick in?
> >
> > 2) I uderstood that activeMQ passes messages to the temp storage area
> when a
> > queue is full 70% (unless configured otherwise), but this is not
> happening
> > here. Why?
> >
> > 3) I've seen a case where each queue is configured with a 500Mb limit,
> > memoryUsage limit is 20Mb and tempUsage is 100Mb, activeMQ accepts 350 Mb
> > worth of messages (70% of 500 Mb) and then it writes it the temporary
> storage,
> > and then it hangs, complaining temp storage is 350% of the maximum value.
> > Then, there's no way to recover activeMQ unless with a restart.
> >
> >
> > Any ideas/suggestions/pointers would be appreciated, as I'm getting a bit
> > frustrated trying to understand this :-)
> >
> > Apologies for the length of the email, hope is easily understandable.
> >
> > Thanks.
> > Regards.
> >
> > --
> > JRF
>
>


-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message