activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shobhana <shobh...@quickride.in>
Subject ActiveMQ broker becomes unresponsive after sometime
Date Thu, 27 Apr 2017 14:05:57 GMT
We use ActiveMQ 5.14.1 with KahaDB for persistence.
>From last 2 days, we have observed that the broker becomes unresponsive
after running for around 4 hours! None of the operations work after this
including establishing new connections, publishing messages to topics,
subscribing to topics, unsubscribing, etc.

We have disabled JMX for performance reasons; so I cannot check the
status/health of broker from JMX. I tried to take thread dump to see what's
happening, but it fails with a message : Unable to open socket file: target
process not responding or HotSpot VM not loaded!

Similar error when I try to take heap dump! But I can see that broker
process is running using ps -ef |grep java option.

Tried to take the thread dump forcefully using the -F option, but this also
fails with "java.lang.RuntimeException: Unable to deduce type of thread from
address 0x00007f9288012800 (expected type JavaThread, CompilerThread,
ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)"

Forceful Heap dump fails with "Expecting GenCollectedHeap, G1CollectedHeap,
or ParallelScavengeHeap, but got sun.jvm.hotspot.gc_interface.CollectedHeap"

We have just one broker running on AWS EC2 Ubuntu instance. Broker is
started with Xmx of 12 GB. Our server and Android applications together
create thousands of topics and exchange MQTT messages (both persistent and
non-persistent). Within 4 hours, around 20 GB of journal files got created
in the last run before broker became unresponsive! The only way to overcome
this problem is to stop the broker, delete all files in KahaDB and restart
the broker!

Any hints to what could be going wrong is highly appreciated!

Broker configurations is given below for reference :

<broker xmlns="http://activemq.apache.org/schema/core" useJmx="false"
brokerName="PrimaryBroker" deleteAllMessagesOnStartup="false"
advisorySupport="false" schedulePeriodForDestinationPurge="600000"
offlineDurableSubscriberTimeout="54000000"
offlineDurableSubscriberTaskSchedule="3600000"
dataDirectory="${activemq.data}">

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" gcInactiveDestinations="true"
inactiveTimoutBeforeGC="3600000">
                  <pendingMessageLimitStrategy>
                    <constantPendingMessageLimitStrategy limit="1000"/>
                  </pendingMessageLimitStrategy>
                  <deadLetterStrategy>
                    <sharedDeadLetterStrategy processExpired="false" />
                  </deadLetterStrategy>
                </policyEntry>
		<policyEntry queue=">" optimizedDispatch="true"
reduceMemoryFootprint="true">
         	  <deadLetterStrategy>
           	    <sharedDeadLetterStrategy processExpired="false" />
         	  </deadLetterStrategy>
		</policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

        <persistenceAdapter>
            <kahaDB directory="${activemq.data}/kahadb"
indexCacheSize="20000" enableJournalDiskSyncs="false"
ignoreMissingJournalfiles="true"/>
        </persistenceAdapter>

          <systemUsage>
            <systemUsage sendFailIfNoSpaceAfterTimeout="10000">
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="50 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector name="openwire"
uri="nio://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxInactivityDuration=180000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="mqtt+nio"
uri="mqtt+nio://0.0.0.0:1883?maximumConnections=50000&amp;wireFormat.maxInactivityDuration=180000&amp;wireFormat.maxFrameSize=104857600"/>
        </transportConnectors>

        <plugins>
            <discardingDLQBrokerPlugin dropAll="true"
dropTemporaryTopics="true" dropTemporaryQueues="true" />
	    <timeStampingBrokerPlugin ttlCeiling="43200000"
zeroExpirationOverride="43200000"/>
        </plugins>

</broker>

TIA,
Shobhana



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-broker-becomes-unresponsive-after-sometime-tp4725278.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message