Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of christian.posta@gmail.com
 designates 209.85.215.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKS7T5M4vmYKb01Gc=tR7NROa3KhZnLUT3oofJoLRDGPsDzGPA@mail.gmail.com>
References: <20130605152432.GB11823@pleuron.we7.local>
	<6755E4F1-E2E8-480D-AD0C-20BD7A2DF00A@gmail.com>
	<CAKS7T5M4vmYKb01Gc=tR7NROa3KhZnLUT3oofJoLRDGPsDzGPA@mail.gmail.com>
Date: Wed, 5 Jun 2013 09:03:15 -0700
Message-ID: 
 <CAKS7T5P2Wf+8RKciaHdY5A7ydw3iZp9u6+BzM3M7YU=G5uASdg@mail.gmail.com>
Subject: Re: Understanding memoryLimits with flow control and systemUsage
 limits.
From: Christian Posta <christian.posta@gmail.com>
To: "users@activemq.apache.org" <users@activemq.apache.org>
Content-Type: multipart/alternative; boundary=14dae9473607e6944f04de6a55cb

--14dae9473607e6944f04de6a55cb
Content-Type: text/plain; charset=ISO-8859-1

was just thinking... for question #3, it would make sense if the overall
broker memory was 500MB and your temp was 100MB... then when you hit the
cursor high water mark at some point (70%) it would spook to disk and
overload the temp... in that case, you have to ensure the temp storage is
sufficient to handle your messages from memory.... but as you described it
(500mb per destination, 20mb overall) then it doesn't work.


On Wed, Jun 5, 2013 at 8:57 AM, Christian Posta
<christian.posta@gmail.com>wrote:

> Sounds like it's happening exactly as you have it configured.. I assume
> you're using non-persistent messages?
>
> As Johan pointed out, your config kinda doesn't make sense :)
>
> The <memoryUsage> is an overall broker limit. That is, if you set your
> <memoryUsage limit="40mb"> then you want to keep the aggregate of all of
> your destinations <= 40mb. Therefore, if you have 40 queues, set each one
> to 1mb. Or some other combo that is <= 40mb. In your case you set each
> queue to 50mb (<policyEntry memoryLimit="50mb"...) which doesn't work well.
> As for your questions...
>
> 1) the 50mb limit doesn't kick in (you don't get PFC) because the broker
> has hit the 70% limit and is spooling your messages to disk. in the first
> case, it never hits the 70% mem limit of your destination because it filled
> up (40mb) the broker main memory first
>
> 2) it sure is happening :) check the logs for scenario #2... you hit the
> limit on the temp store
>
> 3) that doesn't make sense. whip up a test case to show that.
>
> hope that helps :)
>
>
> On Wed, Jun 5, 2013 at 8:39 AM, Johan Edstrom <seijoed@gmail.com> wrote:
>
>> Do it the other way around.
>> The system and store settings are global, the others per destination.
>> So what you are doing doesn't make a ton of sense.
>>
>>
>> On Jun 5, 2013, at 9:24 AM, Jesus Roncero <jesus@we7.com> wrote:
>>
>> > Hi all,
>> >
>> > I've got a testing activeMQ installation that I'm trying to test
>> regarding
>> > different limits on destinations and memory usage in general, with and
>> without
>> > flowControl enable. I'm running into some issues and I don't quite
>> understand
>> > how this works, so I would appreciate any tips or advise here.
>> >
>> > My setup is activeMQ 5.6 in a testing environment. I'm using a couple of
>> > python scripts that use stomp.py to send some messages and read them
>> from a
>> > testqueue. Depending on a couple of settings, the behaviour changes,
>> and it
>> > behaves differently.
>> >
>> > I'm sendind 1 Mb messages to the queue as per:
>> >
>> > data = "x" * 1024 * 1024
>> > number = 1000
>> > print "producing %s %s kb messages" % (number, len(data) / 1024)
>> > for i in range(number):
>> >    print i
>> >    conn.send(data, destination='/queue/testqueue')
>> >
>> > in python.
>> >
>> > Now, I have two settings:
>> >
>> > Scenario number 1
>> > ----------------
>> >
>> > <policyEntries>
>> >     <policyEntry queue=">" memoryLimit="50mb"
>> producerFlowControl="true">
>> >         <pendingQueuePolicy>
>> >             <fileQueueCursor />
>> >         </pendingQueuePolicy>
>> >     </policyEntry>
>> >     <policyEntry topic=">" memoryLimit="50mb"
>> producerFlowControl="true">
>> >         <!-- We want to store both persistent and non-persistent
>> messages -->
>> >         <pendingSubscriberPolicy>
>> >             <fileCursor />
>> >         </pendingSubscriberPolicy>
>> >         <pendingDurableSubscriberPolicy>
>> >             <fileDurableSubscriberCursor/>
>> >         </pendingDurableSubscriberPolicy>
>> >
>> >         ...
>> >
>> >        <systemUsage>
>> >            <systemUsage>
>> >                <memoryUsage>
>> >                    <memoryUsage limit="40 mb"/>
>> >                </memoryUsage>
>> >                <storeUsage>
>> >                    <storeUsage limit="1 gb" />
>> >                </storeUsage>
>> >                <tempUsage>
>> >                    <tempUsage limit="800 mb"/>
>> >                </tempUsage>
>> >            </systemUsage>
>> >        </systemUsage>
>> >
>> > Where the memoryLimit of each of the queues is bigger than the limit for
>> > memoryUsage. When I try to insert into the queue, it insert 32 messages
>> (32Mb)
>> > and it hangs in there. I see this log messages:
>> >
>> > [2013-06-05 15:54:19,816] INFO ActiveMQ Transport: tcp:///
>> 10.0.9.124:36431 org.apache.activemq.broker.region.Queue - Usage Manager
>> Memory Limit (52428800) reached on queue://testqueue. Producers will be
>> throttled to the rate at which messages are removed from this destination
>> to prevent flooding it. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>> > 2013-06-05 15:54:19,816 [0.0.9.124:36431] INFO  Queue  - Usage Manager
>> Memory Limit (52428800) reached on queue://testqueue. Producers will be
>> throttled to the rate at which messages are removed from this destination
>> to prevent flooding it. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>> > [2013-06-05 15:54:20,818] INFO ActiveMQ Transport: tcp:///
>> 10.0.9.124:36431 org.apache.activemq.broker.region.Queue -
>> Usage(default:memory:queue://testqueue:memory) percentUsage=62%,
>> usage=32537724, limit=52428800,
>> percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%,
>> usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager
>> Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1)
>> stopped to prevent flooding queue://testqueue. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>> (blocking for: 1s)
>> > 2013-06-05 15:54:20,818 [0.0.9.124:36431] INFO  Queue  -
>> Usage(default:memory:queue://testqueue:memory) percentUsage=62%,
>> usage=32537724, limit=52428800,
>> percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%,
>> usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager
>> Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1)
>> stopped to prevent flooding queue://testqueue. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>> (blocking for: 1s)
>> >
>> > Which would indicate that flow control kicked in as per the first
>> message,
>> > that they reach a memory limit (50Mb) as per the per destination limit
>> (second
>> > message). It also says that the memory (40Mb) got used as a 100%.
>> >
>> > Also, noted that it says: default:memory:queue.
>> >
>> >
>> > Scenario number 2
>> > -----------------
>> >
>> > Now, if I change the memoryUsage to be slightly bigger than the
>> memoryLimit on
>> > each queue, from 40 to 60Mb, as per:
>> >
>> >        <systemUsage>
>> >            <systemUsage>
>> >                <memoryUsage>
>> >                    <memoryUsage limit="60 mb"/>
>> >                </memoryUsage>
>> >                <storeUsage>
>> >                    <storeUsage limit="1 gb" />
>> >                </storeUsage>
>> >                <tempUsage>
>> >                    <tempUsage limit="800 mb"/>
>> >                </tempUsage>
>> >            </systemUsage>
>> >        </systemUsage>
>> >
>> > taking into account that the per-destination limits are just the same,
>> 50Mb as
>> > per the config above. When I run the same exact procedure, it inserts
>> 801 1Mb
>> > messages and it stops with this message:
>> >
>> > [2013-06-05 16:05:01,972] INFO BrokerService.worker.1
>> org.apache.activemq.store.kahadb.plist.PListStore -
>> PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
>> > 2013-06-05 16:05:01,972 [ervice.worker.1] INFO  PListStore -
>> PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
>> > [2013-06-05 16:05:12,959] INFO ActiveMQ Transport: tcp:///
>> 10.0.9.124:36615 org.apache.activemq.broker.region.Queue -
>> Usage(default:temp:queue://testqueue:temp) percentUsage=99%,
>> usage=839122944, limit=838860800,
>> percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%,
>> usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is
>> Full (99% of 838860800). Stopping producer
>> (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding
>> queue://testqueue. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>> (blocking for: 1s)
>> > 2013-06-05 16:05:12,959 [0.0.9.124:36615] INFO  Queue -
>> Usage(default:temp:queue://testqueue:temp) percentUsage=99%,
>> usage=839122944, limit=838860800,
>> percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%,
>> usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is
>> Full (99% of 838860800). Stopping producer
>> (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding
>> queue://testqueue. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>> (blocking for: 1s)
>> >
>> > Where I see that it starts using the tempUsage space, and it pauses
>> when it's
>> > used the 800 Mb available, whining that the temp store is full.
>> >
>> > Now, the type is default:temp:queue rather than memory
>> >
>> > So,
>> >
>> > Could anyone explain to me why this is? I've read quite a few blog
>> posts and
>> > entries on the mailing list and I'm still a bit confused about it, so,
>> to have
>> > more concrete questions,
>> >
>> > 1) Why this change on the memoryUsage limit affects how activeMQ deals
>> with
>> > way too many messages? More so, why on the second example, the 50 Mb
>> limit
>> > doesn't kick in?
>> >
>> > 2) I uderstood that activeMQ passes messages to the temp storage area
>> when a
>> > queue is full 70% (unless configured otherwise), but this is not
>> happening
>> > here. Why?
>> >
>> > 3) I've seen a case where each queue is configured with a 500Mb limit,
>> > memoryUsage limit is 20Mb and tempUsage is 100Mb, activeMQ accepts 350
>> Mb
>> > worth of messages (70% of 500 Mb) and then it writes it the temporary
>> storage,
>> > and then it hangs, complaining temp storage is 350% of the maximum
>> value.
>> > Then, there's no way to recover activeMQ unless with a restart.
>> >
>> >
>> > Any ideas/suggestions/pointers would be appreciated, as I'm getting a
>> bit
>> > frustrated trying to understand this :-)
>> >
>> > Apologies for the length of the email, hope is easily understandable.
>> >
>> > Thanks.
>> > Regards.
>> >
>> > --
>> > JRF
>>
>>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>


-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

--14dae9473607e6944f04de6a55cb--