[ https://issues.apache.org/jira/browse/AMQ-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151955#comment-13151955
]
Pavel Bekkerman commented on AMQ-3591:
--------------------------------------
I'm yet to try to remove failover: (and leave static: only), but above is our netstat, as
you suggeted.
I don't see any unusual socket states - LISTEN or ESTABLISHED.
Any ideas on how to proceed with the solution?
> bridge transport is stuck on socket write at both ends
> ------------------------------------------------------
>
> Key: AMQ-3591
> URL: https://issues.apache.org/jira/browse/AMQ-3591
> Project: ActiveMQ
> Issue Type: Bug
> Components: Transport
> Affects Versions: 5.5.0
> Environment: 2 redhat linux
> Reporter: Leon Fleysher
>
> We have embedded (into Jboss 7 web app) broker A on one linux connecting through tcp
duplex=true bridge to another standalone broker B. Both brokers exchange messages through
4 common queues. Each broker has its own persistence storage in mssql db.
> After a few hours of flawless work the bridge stops delivering the messages in both directions.
> broker A offending thread stack trace (causing the rest to wait on oneway mutex):
> BrokerService[as2analyticsEmbedded] Task-15 [RUNNABLE, IN_NATIVE] CPU time: 0:00
> java.net.SocketOutputStream.socketWrite0(FileDescriptor, byte[], int, int)
> java.net.SocketOutputStream.socketWrite(byte[], int, int)
> java.net.SocketOutputStream.write(byte[], int, int)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.write(byte[], int, int)
> java.io.DataOutputStream.write(byte[], int, int)
> org.apache.activemq.openwire.v7.BaseDataStreamMarshaller.tightMarshalByteSequence2(ByteSequence,
DataOutput, BooleanStream)
> org.apache.activemq.openwire.v7.MessageMarshaller.tightMarshal2(OpenWireFormat, Object,
DataOutput, BooleanStream)
> org.apache.activemq.openwire.v7.ActiveMQMessageMarshaller.tightMarshal2(OpenWireFormat,
Object, DataOutput, BooleanStream)
> org.apache.activemq.openwire.v7.ActiveMQObjectMessageMarshaller.tightMarshal2(OpenWireFormat,
Object, DataOutput, BooleanStream)
> org.apache.activemq.openwire.OpenWireFormat.marshal(Object, DataOutput)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(Object)
> org.apache.activemq.transport.InactivityMonitor.oneway(Object)
> org.apache.activemq.transport.TransportFilter.oneway(Object)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(Object)
> org.apache.activemq.transport.failover.FailoverTransport.oneway(Object)
> org.apache.activemq.transport.MutexTransport.oneway(Object)
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(Object, ResponseCallback)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(Command)
> org.apache.activemq.network.DemandForwardingBridgeSupport$1.onCommand(Object)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(Object)
> org.apache.activemq.transport.TransportFilter.onCommand(Object)
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport, TransportListener,
Object)
> org.apache.activemq.transport.vm.VMTransport.oneway(Object)
> org.apache.activemq.transport.MutexTransport.oneway(Object)
> org.apache.activemq.transport.ResponseCorrelator.oneway(Object)
> org.apache.activemq.broker.TransportConnection.dispatch(Command)
> org.apache.activemq.broker.TransportConnection.processDispatch(Command)
> org.apache.activemq.broker.TransportConnection.iterate()
> org.apache.activemq.thread.PooledTaskRunner.runTask()
> org.apache.activemq.thread.PooledTaskRunner$1.run()
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable)
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> java.lang.Thread.run()
> broker B offending thread stack trace (causing the rest to wait on oneway mutex):
> ActiveMQ Connection Dispatcher: vm://analyticsCentral#6 [RUNNABLE, IN_NATIVE] CPU time:
0:00
> java.net.SocketOutputStream.socketWrite0(FileDescriptor, byte[], int, int)
> java.net.SocketOutputStream.socketWrite(byte[], int, int)
> java.net.SocketOutputStream.write(byte[], int, int)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush()
> java.io.DataOutputStream.flush()
> org.apache.activemq.transport.tcp.TcpTransport.oneway(Object)
> org.apache.activemq.transport.InactivityMonitor.oneway(Object)
> org.apache.activemq.transport.TransportFilter.oneway(Object)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(Object)
> org.apache.activemq.transport.MutexTransport.oneway(Object)
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(Object, ResponseCallback)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(Command)
> org.apache.activemq.network.DemandForwardingBridgeSupport$1.onCommand(Object)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(Object)
> org.apache.activemq.transport.TransportFilter.onCommand(Object)
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport, TransportListener,
Object)
> org.apache.activemq.transport.vm.VMTransport.oneway(Object)
> org.apache.activemq.transport.MutexTransport.oneway(Object)
> org.apache.activemq.transport.ResponseCorrelator.oneway(Object)
> org.apache.activemq.broker.TransportConnection.dispatch(Command)
> org.apache.activemq.broker.TransportConnection.processDispatch(Command)
> org.apache.activemq.broker.TransportConnection.iterate()
> org.apache.activemq.thread.DedicatedTaskRunner.runTask()
> org.apache.activemq.thread.DedicatedTaskRunner$1.run()
> Both brokers run on a local 1 Gbit network. This scenario happens always after few hours
of work and is recreated consistently.
> Broker A relevant configuration:
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core-5.5.0.xsd"
> >
> <broker xmlns="http://activemq.apache.org/schema/core" brokerName="as2analyticsEmbedded"
persistent="true" dataDirectory="/home/as2/amq_storage" destroyApplicationContextOnStop="true">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1gb">
> <pendingQueuePolicy>
> <fileQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
> <networkConnectors>
> <networkConnector uri="static:(failover:(tcp://10.0.213.38:61616))"
> name="bridge"
> duplex="true"
> conduitSubscriptions="true"
> decreaseNetworkConsumerPriority="false">
> </networkConnector>
> </networkConnectors>
> <!-- both FS and DB must be provided. AMQ will use DB for long-term storage.
> FS contents that has not been consumed yet is stored to DB from time to time
-->
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="/home/as2/amq_storage/as2analyticsEmbedded"
dataSource="#mssql-ds" useDatabaseLock="false">
> <adapter>
> <imageBasedJDBCAdaptor/>
> </adapter>
> </jdbcPersistenceAdapter>
> </persistenceAdapter>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="2gb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="80gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="10gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="as2analyticsEmbeddedConnector" uri="vm://as2analyticsEmbedded"/>
> </transportConnectors>
> </broker>
> <bean id="mssql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
> <property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
> <property name="url" value="jdbc:sqlserver://10.0.213.159:1433;databaseName=obfuscated"/>
> <property name="username" value="obfuscated"/>
> <property name="password" value="obfuscated"/>
> <property name="initialSize" value="1"/>
> <property name="maxActive" value="100"/>
> <property name="poolPreparedStatements" value="true"/>
> </bean>
> </beans>
> Broker B relevant configuration:
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
> <!-- Allows us to use system properties as variables in this configuration file
-->
> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
> -->
> <broker xmlns="http://activemq.apache.org/schema/core" brokerName="analyticsCentral"
dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true">
> <!--
> For better performances use VM cursor and small memory limit.
> For more information, see:
> http://activemq.apache.org/message-cursors.html
> Also, if your producer is "hanging", it's probably due to producer flow control.
> For more information, see:
> http://activemq.apache.org/producer-flow-control.html
> -->
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1gb">
> <pendingQueuePolicy>
> <fileQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> <policyEntry queue=">">
> <deadLetterStrategy>
> <individualDeadLetterStrategy
> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <!--
> The managementContext is used to configure how ActiveMQ is exposed in
> JMX. By default, ActiveMQ uses the MBean server that is started by
> the JVM. For more information, see:
> http://activemq.apache.org/jmx.html
> -->
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
> <!--
> Configure message persistence for the broker. The default persistence
> mechanism is the KahaDB store (identified by the kahaDB tag).
> For more information, see:
> http://activemq.apache.org/persistence.html
> -->
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="${activemq.base}/data" dataSource="#mssql-ds"
useDatabaseLock="false">
> <adapter>
> <imageBasedJDBCAdaptor/>
> </adapter>
> </jdbcPersistenceAdapter>
> </persistenceAdapter>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="2gb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="10gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="5gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <!--
> The transport connectors expose ActiveMQ over a given protocol to
> clients and other brokers. For more information, see:
> http://activemq.apache.org/configuring-transports.html
> -->
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
> </transportConnectors>
> </broker>
> <amq:connectionFactory id="jmsFactory" brokerURL="tcp://localhost">
> <amq:redeliveryPolicy>
> <amq:redeliveryPolicy maximumRedeliveries="0"/>
> </amq:redeliveryPolicy>
> </amq:connectionFactory>
> <bean id="mssql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
> <property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
> <property name="url" value="jdbc:sqlserver://10.0.213.159:1433;DatabaseName=obfuscated"/>
> <property name="username" value="obfuscated"/>
> <property name="password" value="obfuscated"/>
> <property name="initialSize" value="1"/>
> <property name="maxActive" value="100"/>
> <property name="poolPreparedStatements" value="true"/>
> </bean>
> <!--
> Enable web consoles, REST and Ajax APIs and demos
> It also includes Camel (with its web console), see ${ACTIVEMQ_HOME}/conf/camel.xml
for more info
> Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
> -->
> <import resource="jetty.xml"/>
> </beans>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
|