Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Brian Tanner <btanner@edmunds.com>
Content-Type: multipart/alternative; boundary="Apple-Mail-1-826527192"
Subject: Disk store usage with virtual topics
Date: Fri, 19 Mar 2010 17:13:39 -0700
Message-ID: <DE35FCF9-04B1-4E0A-A8AB-BC78E0747AE9@edmunds.com>
To: <users@activemq.apache.org>
MIME-Version: 1.0 (Apple Message framework v1077)

--Apple-Mail-1-826527192
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="us-ascii"


Hi, our development team is observing a lot of storage retention when =
using virtual topics under the following scenario (we are using the =
default store cursors with kahadb persistence, activemq.xml provided =
below):

We create one virtual topic with a single queue wired to it.  When we =
send a batch of messages, with no consumers on the queue and observe a =
certain amount of store usage, call it 100mb.  We turn on a consumer for =
the queue and drain the messages. Storage usage falls close to zero.  =
This makes sense to everybody.  Now we add two more queues to the same =
virtual topic, remove all consumers, then run another batch of messages. =
 Store usage spikes up to 300mb.  This suggests each queue gets its own =
complete copy of the message, which is fine, we can tolerate that.  We =
turn on consumers for all queues, drain the messages from the queues and =
our disk store usage returns close to zero.  Everything is as expected.  =
Now the problem scenario:  We  run a third batch of messages, again with =
no consumers on.  Disk store goes to 300mb.  Then we turn consumers on =
for TWO of the THREE queues on the virtual topic.  Two of the queues are =
drained, but the third queue is still full.  We would expect disk store =
usage to be 100mb reflecting the one full queue's worth of messages.  =
Instead disk store stays at 300mb. =20

This suggests that when messages are multiplexed through a virtual =
topic, the disk store will not release space for a message until ALL =
queues on the virtual topic have consumed this message.  Is this =
expected behavior?  If so, why does disk store allocate 3x space for 3x =
queues if they seem to be tied to the same life cycle?  Can anyone help =
explain?

Our config running on activemq (fuse branch) 5.3.0.5:

<beans xmlns=3D"http://www.springframework.org/schema/beans" =
xmlns:amq=3D"http://activemq.apache.org/schema/core" =
xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance" =
xsi:schemaLocation=3D"http://www.springframework.org/schema/beans =
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd =
http://activemq.apache.org/schema/core =
http://activemq.apache.org/schema/core/activemq-core.xsd">
   <!-- Allows us to use system properties as variables in this =
configuration file -->
   <bean =
class=3D"org.springframework.beans.factory.config.PropertyPlaceholderConfi=
gurer">
       <property name=3D"locations">
           =
<value>file:${activemq.base}/conf/credentials.properties</value>
       </property>
   </bean>
<broker xmlns=3D"http://activemq.apache.org/schema/core" =
brokerName=3D"${broker.name}" useJmx=3D"true" =
dataDirectory=3D"/content/activemq/amq-data/${broker.name}/persist-store-$=
{broker.name}" persistenceAdapter=3D"persist-store">
       <destinationPolicy>
           <policyMap>
               <policyEntries>
                   <policyEntry topic=3D">" producerFlowControl=3D"false" =
memoryLimit=3D"5mb" />
                   <policyEntry queue=3D">" producerFlowControl=3D"false" =
memoryLimit=3D"5mb" />
               </policyEntries>
           </policyMap>
       </destinationPolicy>
       <managementContext>
           <managementContext connectorPort=3D"1101" =
rmiServerPort=3D"1100" jmxDomainName=3D"org.apache.activemq"/>
       </managementContext>
       <persistenceAdapter>
            <KahaDB =
directory=3D"/content/activemq/amq-data/${broker.name}/kahadb-${broker.nam=
e}" enableIndexWriteAsync=3D"true" enableJournalDiskSyncs=3D"false" =
indexWriteBatchSize=3D"1000" checkpointInterval=3D"5000"  =
cleanupInterval=3D"5000"/>
       </persistenceAdapter>
       <transportConnectors>
           <transportConnector name=3D"nio" uri=3D"nio://0.0.0.0:61616"/>
       </transportConnectors>
       <systemUsage>
         <systemUsage>
             <memoryUsage>
                 <memoryUsage limit=3D"20 mb"/>
             </memoryUsage>
             <storeUsage>
                 <storeUsage limit=3D"4 gb" name=3D"persist-store"/>
             </storeUsage>
             <tempUsage>
                 <tempUsage limit=3D"100 mb"/>
             </tempUsage>
         </systemUsage>
     </systemUsage>
   </broker>
   <import resource=3D"jetty.xml"/>
</beans>=

--Apple-Mail-1-826527192--