activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: TempQueue advisory flood
Date Thu, 17 May 2012 04:27:49 GMT
sorry, yes, there is one more piece to the puzzle. The temp
destinations need to have a well known naming convention and there
needs to be a wildcard static include in the network bridge to have
matching destinations propagated.

Have a look at the test case to see an example of the xml config required:
http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/RequestReplyNoAdvisoryNetworkTest.java

In that code, note the use of:
factory.setConnectionIDPrefix(...) to specify a prefix for all temp
destinations that can be used for the wildcard static include
<staticallyIncludedDestinations>

This is needed because the destination advisories are no longer propagated.

On 16 May 2012 18:13, Chris Robison <chrisdrobison@gmail.com> wrote:
> So I tried that and the client appears to be sending just fine, but the
> producer on the other side of the network connector that expects to hear
> back via the temp queue is not receiving the message.
>
> Chris
>
> On Wed, May 16, 2012 at 1:27 AM, Gary Tully <gary.tully@gmail.com> wrote:
>
>> you need one additional bit of config, to disable the connection from
>> tracking temp destination advisories so that it allows the request to
>> go to the broker.
>>
>> factory.setWatchTopicAdvisories(false);
>> or ?jms.watchTopicAdvisories=false on the connection factory broker url
>>
>> On 15 May 2012 23:30, Chris Robison <chrisdrobison@gmail.com> wrote:
>> > I tried that last suggestion and it appears as though
>> > the allowTempAutoCreationOnSend is not working. The client keep
>> receiving a
>> > "cannot send to a deleted destination" error.
>> >
>> > Chris
>> >
>> > On Mon, May 14, 2012 at 4:19 PM, Gary Tully <gary.tully@gmail.com>
>> wrote:
>> >
>> >> this is expected, but the flooding impact should be minimal with a
>> >> small number of brokers in the network and a reasonable number of
>> >> connections.
>> >>
>> >> each network bridge will register interest in all temp destination
>> >> advisories, so that it will be in a position to receive messages for
>> >> those destinations. It can't know in advance which messages may be
>> >> routed though it, so it needs to react to each advisory message and
>> >> create a local temp destination. When the connection that creates the
>> >> temp is closes, the temp destination gets closed and each broker
>> >> reacts to that event via the remove temp advisory.
>> >>
>> >> Temp destinations are often pooled for serial request reply messages
>> >> or multiplexed using selectors, to reduce the overhead of creation and
>> >> propagation, in a network scenario. The other option is to use regular
>> >> destinations for replys
>> >>
>> >> For temp destinations, there is an alternative solution that would
>> >> require a little more configuration but which would reduce the
>> >> overhead of unnecessary temp destination propagation.
>> >> You can disable auto temp destination propagation through the network
>> >> bridge, ie: bridgeTempDestinations=false.
>> >> Then you need to allow temp destination creation on the fly for a
>> >> producer, broker.allowTempAutoCreationOnSend=true
>> >> To finish, you need to configure
>> >> broker.schedulePeriodForDestinationPurge>0 to have the on the fly
>> >> created temp destinations purged when they are empty and enable it for
>> >> temp destinations. policy entry gcInactiveDestinations=true (this is
>> >> important because the auto created temp destination cannot be
>> >> associated with a connection in any sensible way)
>> >>
>> >> There is a test that exercises these options
>> >>
>> >>
>> http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/RequestReplyNoAdvisoryNetworkTest.java
>> >>
>> >>
>> >> On 11 May 2012 17:45, Chris Robison <chrisdrobison@gmail.com> wrote:
>> >> > Another interesting observation: as a test, I connected all producers
>> and
>> >> > consumers to MSTMIP102 (the machine that doesn't have the network
>> >> > connector). I then started MSIPAP101 so that its network connector
>> could
>> >> > start and connect to MSTMIP102. Request request-reply works fine
>> between
>> >> > all producers and consumers, but after 1 or 2 request-replies,
>> MSIPAP101
>> >> is
>> >> > flooded with the temp queue advisories--even though there are no
>> >> producers
>> >> > or consumers connected to it.
>> >> >
>> >> > On Fri, May 11, 2012 at 10:09 AM, Chris Robison <
>> chrisdrobison@gmail.com
>> >> >wrote:
>> >> >
>> >> >> Here is the log from the latest run with the TTL down to 1. It still
>> >> >> floods after about 1 or 2 request-reply calls. It just seems to be
>> >> >> alternating between add temp queue and remove temp queue messages.
>> >> >>
>> >> >> On Fri, May 11, 2012 at 9:52 AM, Chris Robison <
>> chrisdrobison@gmail.com
>> >> >wrote:
>> >> >>
>> >> >>> Yes, because we have a spoke and hub topology where a client in one
>> >> spoke
>> >> >>> may need to request info from a client connected to a different
>> spoke.
>> >> I
>> >> >>> suppose I could lower that value.
>> >> >>>
>> >> >>>
>> >> >>> On Fri, May 11, 2012 at 9:47 AM, Gary Tully <gary.tully@gmail.com>
>> >> wrote:
>> >> >>>
>> >> >>>> Do you have a good reason to have networkTTL="5", if there is only
>> one
>> >> >>>> broker, the default value of 1 will be fine.
>> >> >>>>
>> >> >>>> On 11 May 2012 16:32, Chris Robison <chrisdrobison@gmail.com>
>> wrote:
>> >> >>>> > If it helps, I've attached the configuration files for both
>> >> machines.
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > On Fri, May 11, 2012 at 9:07 AM, Chris Robison <
>> >> >>>> chrisdrobison@gmail.com>
>> >> >>>> > wrote:
>> >> >>>> >>
>> >> >>>> >> One thing I have noticed though is that when I restart the
>> broker
>> >> on
>> >> >>>> the
>> >> >>>> >> other end, everything starts to work again until it floods
>> again.
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>> >> On Fri, May 11, 2012 at 9:05 AM, Chris Robison <
>> >> >>>> chrisdrobison@gmail.com>
>> >> >>>> >> wrote:
>> >> >>>> >>>
>> >> >>>> >>> I'm on 5.6. And all brokers remain up and running.
>> >> >>>> >>>
>> >> >>>> >>>
>> >> >>>> >>> On Fri, May 11, 2012 at 8:48 AM, Gary Tully <
>> gary.tully@gmail.com
>> >> >
>> >> >>>> wrote:
>> >> >>>> >>>>
>> >> >>>> >>>> what version are you on?
>> >> >>>> >>>> Is there any chance that the broker at the other end of the
>> >> network
>> >> >>>> >>>> bridge is shutting down?
>> >> >>>> >>>>
>> >> >>>> >>>> On 11 May 2012 14:34, Chris Robison <chrisdrobison@gmail.com>
>> >> >>>> wrote:
>> >> >>>> >>>> > I have a network of brokers and I am using a request-reply
>> >> system
>> >> >>>> >>>> > across
>> >> >>>> >>>> > that network, but I've noticed that after a few
>> request-replys,
>> >> >>>> the
>> >> >>>> >>>> > broker
>> >> >>>> >>>> > network will get flooded with TempQueue advisory messages,
>> then
>> >> >>>> some
>> >> >>>> >>>> > of the
>> >> >>>> >>>> > of the brokers will be reported as slow consumers and the
>> >> >>>> consumers
>> >> >>>> >>>> > won't
>> >> >>>> >>>> > receive messages after that. The advisory messages seem to
>> be
>> >> >>>> repeated
>> >> >>>> >>>> > (e.g. add temp destination, remove temp destination) for the
>> >> same
>> >> >>>> temp
>> >> >>>> >>>> > destinations. Not quite sure how to handle this. Any
>> >> suggestions?
>> >> >>>> >>>> >
>> >> >>>> >>>> > Here are some of the errors are other things I'm seeing
>> >> >>>> repeatedly in
>> >> >>>> >>>> > logs
>> >> >>>> >>>> > when I enable debug. In fact, when this flood occurs, tons
>> of
>> >> log
>> >> >>>> gets
>> >> >>>> >>>> > generated all at once.
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 05:24:37,248 | DEBUG | Error occured while
>> >> processing
>> >> >>>> async
>> >> >>>> >>>> > command: DestinationInfo {commandId = 40578,
>> responseRequired =
>> >> >>>> false,
>> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2,
>> >> destination =
>> >> >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
>> >> >>>> >>>> > operationType =
>> >> >>>> >>>> > 0, timeout = 0, brokerPath =
>> >> >>>> [ID:MSSD5D101-38580-1336708634042-0:1]},
>> >> >>>> >>>> > exception: java.lang.IllegalStateException: Timer already
>> >> >>>> cancelled. |
>> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
>> >> ActiveMQ
>> >> >>>> >>>> > Transport: ssl:///10.60.252.28:54493
>> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled.
>> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397)
>> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
>> >> >>>> >>>> > at
>> >> org.apache.activemq.broker.region.Queue.start(Queue.java:889)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 05:24:37,248 | WARN  | Async error occurred:
>> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. |
>> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
>> >> ActiveMQ
>> >> >>>> >>>> > Transport: ssl:///10.60.252.28:54493
>> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled.
>> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397)
>> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
>> >> >>>> >>>> > at
>> >> org.apache.activemq.broker.region.Queue.start(Queue.java:889)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 05:24:37,249 | DEBUG | mstmip102.vosh.net adding
>> >> >>>> >>>> > destination:
>> >> >>>> >>>> > topic://ActiveMQ.Advisory.Connection |
>> >> >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion |
>> >> StopAsync:vm://
>> >> >>>> >>>> > mstmip102.vosh.net#0-23
>> >> >>>> >>>> > 2012-05-11 05:24:37,250 | WARN  | Failed to remove
>> connection
>> >> >>>> >>>> > ConnectionInfo {commandId = 3, responseRequired = false,
>> >> >>>> connectionId
>> >> >>>> >>>> > =
>> >> >>>> >>>> > ID:msipap102-50908-1336690011745-7:1, clientId =
>> >> >>>> >>>> > mstmip102_msipap102.vosh.net_outbound, clientIp =
>> >> >>>> >>>> > vm://mstmip102.vosh.net#0,
>> >> >>>> >>>> > userName = mqipauser, password = *****, brokerPath = null,
>> >> >>>> >>>> > brokerMasterConnector = false, manageable = false,
>> >> clientMaster =
>> >> >>>> >>>> > true,
>> >> >>>> >>>> > faultTolerant = false, failoverReconnect = false}, reason:
>> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. |
>> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
>> >> >>>> >>>> > StopAsync:vm://
>> >> >>>> >>>> > mstmip102.vosh.net#0-23
>> >> >>>> >>>> > 2012-05-11 05:24:37,250 | DEBUG | Exception detail: |
>> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
>> >> >>>> >>>> > StopAsync:vm://
>> >> >>>> >>>> > mstmip102.vosh.net#0-23
>> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled.
>> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397)
>> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
>> >> >>>> >>>> > at
>> >> org.apache.activemq.broker.region.Topic.start(Topic.java:530)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:286)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:481)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:533)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:464)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:459)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConnection(AdvisoryBroker.java:234)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:351)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.security.JaasAuthenticationBroker.removeConnection(JaasAuthenticationBroker.java:95)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:119)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:766)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068)
>> >> >>>> >>>> > at
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>>
>> >>
>> org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996)
>> >> >>>> >>>> > at java.lang.Thread.run(Thread.java:722)
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 05:24:37,252 | DEBUG | mstmip102.vosh.net adding
>> >> >>>> >>>> > destination:
>> >> >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1 |
>> >> >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion | ActiveMQ
>> >> >>>> Transport:
>> >> >>>> >>>> > ssl:///172.17.5.78:59658
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging
>> (mstmip102.vosh.net->
>> >> >>>> >>>> > msipap101.vosh.net)
>> >> >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184,
>> >> >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1,
>> destination
>> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath:
>> >> >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message:
>> >> ActiveMQMessage
>> >> >>>> >>>> > {commandId
>> >> >>>> >>>> > = 0, responseRequired = false, messageId =
>> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184,
>> >> >>>> originalDestination =
>> >> >>>> >>>> > null,
>> >> >>>> >>>> > originalTransactionId = null, producerId =
>> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination =
>> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = null,
>> >> >>>> expiration
>> >> >>>> >>>> > = 0,
>> >> >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588,
>> >> >>>> >>>> > brokerOutTime =
>> >> >>>> >>>> > 1336713877341, correlationId = null, replyTo = null,
>> >> persistent =
>> >> >>>> >>>> > false,
>> >> >>>> >>>> > type = Advisory, priority = 0, groupID = null,
>> groupSequence =
>> >> 0,
>> >> >>>> >>>> > targetConsumerId = null, compressed = false, userID = null,
>> >> >>>> content =
>> >> >>>> >>>> > null,
>> >> >>>> >>>> > marshalledProperties =
>> >> >>>> org.apache.activemq.util.ByteSequence@567edec8,
>> >> >>>> >>>> > dataStructure = DestinationInfo {commandId = 0,
>> >> responseRequired =
>> >> >>>> >>>> > false,
>> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2,
>> >> destination =
>> >> >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
>> >> >>>> >>>> > operationType =
>> >> >>>> >>>> > 0, timeout = 0, brokerPath = null}, redeliveryCounter = 0,
>> >> size =
>> >> >>>> >>>> > 1024,
>> >> >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616,
>> >> >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1,
>> >> >>>> originBrokerName=
>> >> >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false,
>> readOnlyBody
>> >> =
>> >> >>>> false,
>> >> >>>> >>>> > droppable = false} |
>> >> >>>> >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport |
>> >> >>>> ActiveMQ
>> >> >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging
>> (mstmip102.vosh.net->
>> >> >>>> >>>> > msipap101.vosh.net)
>> >> >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185,
>> >> >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1,
>> destination
>> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath:
>> >> >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message:
>> >> ActiveMQMessage
>> >> >>>> >>>> > {commandId
>> >> >>>> >>>> > = 0, responseRequired = false, messageId =
>> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185,
>> >> >>>> originalDestination =
>> >> >>>> >>>> > null,
>> >> >>>> >>>> > originalTransactionId = null, producerId =
>> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination =
>> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = null,
>> >> >>>> expiration
>> >> >>>> >>>> > = 0,
>> >> >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588,
>> >> >>>> >>>> > brokerOutTime =
>> >> >>>> >>>> > 1336713877341, correlationId = null, replyTo = null,
>> >> persistent =
>> >> >>>> >>>> > false,
>> >> >>>> >>>> > type = Advisory, priority = 0, groupID = null,
>> groupSequence =
>> >> 0,
>> >> >>>> >>>> > targetConsumerId = null, compressed = false, userID = null,
>> >> >>>> content =
>> >> >>>> >>>> > null,
>> >> >>>> >>>> > marshalledProperties =
>> >> >>>> org.apache.activemq.util.ByteSequence@235448d5,
>> >> >>>> >>>> > dataStructure = DestinationInfo {commandId = 0,
>> >> responseRequired =
>> >> >>>> >>>> > false,
>> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2,
>> >> destination =
>> >> >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
>> >> >>>> >>>> > operationType =
>> >> >>>> >>>> > 1, timeout = 0, brokerPath = null}, redeliveryCounter = 0,
>> >> size =
>> >> >>>> >>>> > 1024,
>> >> >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616,
>> >> >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1,
>> >> >>>> originBrokerName=
>> >> >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false,
>> readOnlyBody
>> >> =
>> >> >>>> false,
>> >> >>>> >>>> > droppable = false} |
>> >> >>>> >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport |
>> >> >>>> ActiveMQ
>> >> >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 13:29:40,053 | WARN  | TopicSubscription:
>> >> >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-4:2:1:1,
>> >> destinations=7,
>> >> >>>> >>>> > dispatched=1000, delivered=7723, matched=1001, discarded=0:
>> has
>> >> >>>> twice
>> >> >>>> >>>> > its
>> >> >>>> >>>> > prefetch limit pending, without an ack; it appears to be
>> slow |
>> >> >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription |
>> ActiveMQ
>> >> >>>> >>>> > Transport:
>> >> >>>> >>>> > ssl:///10.60.252.28:54614
>> >> >>>> >>>> >
>> >> >>>> >>>> > 2012-05-11 13:29:42,540 | WARN  | TopicSubscription:
>> >> >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-3:2:1:1,
>> >> destinations=6,
>> >> >>>> >>>> > dispatched=1000, delivered=9922, matched=1001, discarded=0:
>> has
>> >> >>>> twice
>> >> >>>> >>>> > its
>> >> >>>> >>>> > prefetch limit pending, without an ack; it appears to be
>> slow |
>> >> >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription |
>> ActiveMQ
>> >> >>>> >>>> > Transport:
>> >> >>>> >>>> > ssl:///10.60.252.28:54614
>> >> >>>> >>>>
>> >> >>>> >>>>
>> >> >>>> >>>>
>> >> >>>> >>>> --
>> >> >>>> >>>> http://fusesource.com
>> >> >>>> >>>> http://blog.garytully.com
>> >> >>>> >>>
>> >> >>>> >>>
>> >> >>>> >>
>> >> >>>> >
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>> http://fusesource.com
>> >> >>>> http://blog.garytully.com
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>
>> >>
>> >>
>> >>
>> >> --
>> >> http://fusesource.com
>> >> http://blog.garytully.com
>> >>
>>
>>
>>
>> --
>> http://fusesource.com
>> http://blog.garytully.com
>>



-- 
http://fusesource.com
http://blog.garytully.com

Mime
View raw message