activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From someword <derek.ol...@jivesoftware.com>
Subject Broker to Broker prefetch problems
Date Thu, 27 Jun 2013 16:28:37 GMT
Hello.
activemq-5.7.0-1, java version "1.6.0_24".
We have 16 activemq nodes in a mesh network.   The only current application
using the activemq network is MCollective.

I'm very new to Activemq and am hoping to get some pointers as to wether I
am going down the right path in troubleshooting the symptoms we are seeing.  
>From the client application point of view we are seeing the following
symptoms.  Without getting into too much details about what mcollective is
let me generically describe the symptoms like this.

- We do not see responses from the nodes connected to 1 of the brokers 
- Requests to some nodes never make it to the destination node
- Responses from some nodes never make it back to the client

On all of the brokers we see these log messages from activemq 
2013-06-27 15:51:03,608 [160.70.27:57274] WARN  TopicSubscription             
- TopicSubscription: consumer=ID:mco-broker8-57609-1372346502469-88:1:1:1,
destinations=144, dispatched=1000, delivered=0, matched=1001, discarded=0:
has twice its prefetch limit pending, without an ack; it appears to be slow

We have adjusted the prefetch from the default of 1000 to 5000 and 15000
without any change.   I set optimizeAcknowledge = true without any obvious
benefits.   I can't actually see via the attribute optimizeAcknowledge when
I browse via jmx so while it's in the xml file it's possible I put it in the
wrong spot?


Basically i'm hoping someone can tell me that yes the warnings above are
related to the symptoms i've described and I should focus my efforts on
trying to understand/fix whatever is causing this.

What's strange is that these log messages occur even  when the only activity
going on with mcollective is essentially just a heartbeat from each node
(10K nodes) every 5 minutes. 

Any ideas on how I can move forward with figuring out what the problem is. 
I'm figuring I just don't understand activemq enough and have some
configurations which are causing this situation.   

I've included an activemq.xml here.   IT's the same on all 16 nodes.  I've
stripped out the auth section.
TIA!  Derek.


<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
  http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">

    <bean class="org.apache.activemq.spring.ActiveMQConnectionFactory">
        <property name="optimizeAcknowledge">
            <value>true</value>
        </property>
    </bean>

    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.base}/conf/credentials.properties</value>
        </property>
    </bean>

    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="mco-broker2.dc1.domain.name" useJmx="true"
schedulePeriodForDestinationPurge="60000" networkConnectorStartAsync="true"
deleteAllMessagesOnStartup="true">
        <destinationPolicy>
          <policyMap>
            <policyEntries>
              <policyEntry topic=">" producerFlowControl="false"/>
              <policyEntry queue="*.reply.>" gcInactiveDestinations="true"
inactiveTimoutBeforeGC="300000" />
            </policyEntries>
          </policyMap>
        </destinationPolicy>

        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>

        <networkConnectors>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker1.dc1.domain.name-topics"
                uri="static:(ssl://mco-broker1.dc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker1.dc1.domain.name-queues"
                uri="static:(ssl://mco-broker1.dc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker1.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker1.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker1.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker1.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker10.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker10.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker10.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker10.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker11.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker11.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker11.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker11.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker12.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker12.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker12.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker12.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker2.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker2.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker2.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker2.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker3.dc1.domain.name-topics"
                uri="static:(ssl://mco-broker3.dc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker3.dc1.domain.name-queues"
                uri="static:(ssl://mco-broker3.dc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker3.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker3.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker3.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker3.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker4.dc1.domain.name-topics"
                uri="static:(ssl://mco-broker4.dc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker4.dc1.domain.name-queues"
                uri="static:(ssl://mco-broker4.dc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker4.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker4.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker4.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker4.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker5.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker5.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker5.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker5.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker6.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker6.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker6.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker6.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker7.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker7.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker7.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker7.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker8.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker8.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker8.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker8.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker9.bc1.domain.name-topics"
                uri="static:(ssl://mco-broker9.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                prefetchSize="15000"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true">
                <excludedDestinations>
                        <queue physicalName=">" />
                </excludedDestinations>
          </networkConnector>
          <networkConnector
               
name="mco-broker2.dc1.domain.name-mco-broker9.bc1.domain.name-queues"
                uri="static:(ssl://mco-broker9.bc1.domain.name:6167)"
                userName="XXXX"
                password="XXX"
                duplex="false"
                decreaseNetworkConsumerPriority="true"
                networkTTL="1"
                dynamicOnly="true"
                conduitSubscriptions="false">
                <excludedDestinations>
                        <topic physicalName=">" />
                </excludedDestinations>
          </networkConnector>
        </networkConnectors>

        <persistenceAdapter>
          <kahaDB directory="/var/cache/activemq/data"
journalMaxFileLength="64mb"/>
        </persistenceAdapter>

        <plugins>
          <statisticsBrokerPlugin/>
          <simpleAuthenticationPlugin>
          </simpleAuthenticationPlugin>
          <authorizationPlugin>
            <map>
              <authorizationMap>
                <authorizationEntries>
                  <authorizationEntry queue=">" write="admins" read="admins"
admin="admins" />
                  <authorizationEntry queue="global.reply.>"
write="allcollectives" admin="allcollectives" />
                  <authorizationEntry queue="eu.reply.>" write="eu"
admin="eu" />
                  <authorizationEntry queue="eu_infra.reply.>"
write="eu_infra" admin="eu_infra" />
                  <authorizationEntry queue="eu_services.reply.>"
write="eu_services" admin="eu_services" />
                  <authorizationEntry queue="eu_customer.reply.>"
write="eu_customer" admin="eu_customer" />
                  <authorizationEntry queue="global.nodes"
read="allcollectives" admin="allcollectives" />
                  <authorizationEntry queue="eu.nodes" read="eu" admin="eu"
/>
                  <authorizationEntry queue="eu_infra.nodes" read="eu_infra"
admin="eu_infra" />
                  <authorizationEntry queue="eu_services.nodes"
read="eu_services" admin="eu_services" />
                  <authorizationEntry queue="eu_customer.nodes"
read="eu_customer" admin="eu_customer" />
                  <authorizationEntry topic=">" write="admins" read="admins"
admin="admins" />
                  <authorizationEntry topic="global.registration.agent"
write="allcollectives" read="allcollectives" admin="allcollectives" />
                  <authorizationEntry topic="global.*.agent"
read="allcollectives" admin="allcollectives" />
                  <authorizationEntry topic="eu.registration.agent"
write="eu" read="eu" admin="eu" />
                  <authorizationEntry topic="eu.*.agent" read="eu"
admin="eu" />
                  <authorizationEntry topic="eu_infra.*.agent"
read="eu_infra" admin="eu_infra" />
                  <authorizationEntry topic="eu_services.*.agent"
read="eu_services" admin="eu_services" />
                  <authorizationEntry topic="eu_customer.*.agent"
read="eu_customer" admin="eu_customer" />
                  <authorizationEntry topic="ActiveMQ.Advisory.>"
write="everyone" read="everyone" admin="everyone" />
                </authorizationEntries>
              </authorizationMap>
            </map>
          </authorizationPlugin>
        </plugins>

        <sslContext>
          <sslContext
          />
        </sslContext>

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="2 gb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="10 gb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="1 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector name="openwiressl"
uri="ssl://0.0.0.0:6167"/>
            <transportConnector name="stompssl"
uri="stomp+ssl://0.0.0.0:6164"/>
        </transportConnectors>

    </broker>
    <import resource="jetty.xml"/>
</beans>





--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-to-Broker-prefetch-problems-tp4668660.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message