activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Robison <chrisdrobi...@gmail.com>
Subject Re: TempQueue advisory flood
Date Thu, 17 May 2012 22:32:55 GMT
I've been able to successfully reproduce the issue in a dev environment,
but it's hard to identity exactly what is causing it. It is almost like it
is a race condition of some kind.

On Wed, May 16, 2012 at 10:27 PM, Gary Tully <gary.tully@gmail.com> wrote:

> sorry, yes, there is one more piece to the puzzle. The temp
> destinations need to have a well known naming convention and there
> needs to be a wildcard static include in the network bridge to have
> matching destinations propagated.
>
> Have a look at the test case to see an example of the xml config required:
>
> http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/RequestReplyNoAdvisoryNetworkTest.java
>
> In that code, note the use of:
> factory.setConnectionIDPrefix(...) to specify a prefix for all temp
> destinations that can be used for the wildcard static include
> <staticallyIncludedDestinations>
>
> This is needed because the destination advisories are no longer propagated.
>
> On 16 May 2012 18:13, Chris Robison <chrisdrobison@gmail.com> wrote:
> > So I tried that and the client appears to be sending just fine, but the
> > producer on the other side of the network connector that expects to hear
> > back via the temp queue is not receiving the message.
> >
> > Chris
> >
> > On Wed, May 16, 2012 at 1:27 AM, Gary Tully <gary.tully@gmail.com>
> wrote:
> >
> >> you need one additional bit of config, to disable the connection from
> >> tracking temp destination advisories so that it allows the request to
> >> go to the broker.
> >>
> >> factory.setWatchTopicAdvisories(false);
> >> or ?jms.watchTopicAdvisories=false on the connection factory broker url
> >>
> >> On 15 May 2012 23:30, Chris Robison <chrisdrobison@gmail.com> wrote:
> >> > I tried that last suggestion and it appears as though
> >> > the allowTempAutoCreationOnSend is not working. The client keep
> >> receiving a
> >> > "cannot send to a deleted destination" error.
> >> >
> >> > Chris
> >> >
> >> > On Mon, May 14, 2012 at 4:19 PM, Gary Tully <gary.tully@gmail.com>
> >> wrote:
> >> >
> >> >> this is expected, but the flooding impact should be minimal with a
> >> >> small number of brokers in the network and a reasonable number of
> >> >> connections.
> >> >>
> >> >> each network bridge will register interest in all temp destination
> >> >> advisories, so that it will be in a position to receive messages for
> >> >> those destinations. It can't know in advance which messages may be
> >> >> routed though it, so it needs to react to each advisory message and
> >> >> create a local temp destination. When the connection that creates the
> >> >> temp is closes, the temp destination gets closed and each broker
> >> >> reacts to that event via the remove temp advisory.
> >> >>
> >> >> Temp destinations are often pooled for serial request reply messages
> >> >> or multiplexed using selectors, to reduce the overhead of creation
> and
> >> >> propagation, in a network scenario. The other option is to use
> regular
> >> >> destinations for replys
> >> >>
> >> >> For temp destinations, there is an alternative solution that would
> >> >> require a little more configuration but which would reduce the
> >> >> overhead of unnecessary temp destination propagation.
> >> >> You can disable auto temp destination propagation through the network
> >> >> bridge, ie: bridgeTempDestinations=false.
> >> >> Then you need to allow temp destination creation on the fly for a
> >> >> producer, broker.allowTempAutoCreationOnSend=true
> >> >> To finish, you need to configure
> >> >> broker.schedulePeriodForDestinationPurge>0 to have the on the fly
> >> >> created temp destinations purged when they are empty and enable it
> for
> >> >> temp destinations. policy entry gcInactiveDestinations=true (this is
> >> >> important because the auto created temp destination cannot be
> >> >> associated with a connection in any sensible way)
> >> >>
> >> >> There is a test that exercises these options
> >> >>
> >> >>
> >>
> http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/RequestReplyNoAdvisoryNetworkTest.java
> >> >>
> >> >>
> >> >> On 11 May 2012 17:45, Chris Robison <chrisdrobison@gmail.com> wrote:
> >> >> > Another interesting observation: as a test, I connected all
> producers
> >> and
> >> >> > consumers to MSTMIP102 (the machine that doesn't have the network
> >> >> > connector). I then started MSIPAP101 so that its network connector
> >> could
> >> >> > start and connect to MSTMIP102. Request request-reply works fine
> >> between
> >> >> > all producers and consumers, but after 1 or 2 request-replies,
> >> MSIPAP101
> >> >> is
> >> >> > flooded with the temp queue advisories--even though there are no
> >> >> producers
> >> >> > or consumers connected to it.
> >> >> >
> >> >> > On Fri, May 11, 2012 at 10:09 AM, Chris Robison <
> >> chrisdrobison@gmail.com
> >> >> >wrote:
> >> >> >
> >> >> >> Here is the log from the latest run with the TTL down to 1. It
> still
> >> >> >> floods after about 1 or 2 request-reply calls. It just seems to be
> >> >> >> alternating between add temp queue and remove temp queue messages.
> >> >> >>
> >> >> >> On Fri, May 11, 2012 at 9:52 AM, Chris Robison <
> >> chrisdrobison@gmail.com
> >> >> >wrote:
> >> >> >>
> >> >> >>> Yes, because we have a spoke and hub topology where a client in
> one
> >> >> spoke
> >> >> >>> may need to request info from a client connected to a different
> >> spoke.
> >> >> I
> >> >> >>> suppose I could lower that value.
> >> >> >>>
> >> >> >>>
> >> >> >>> On Fri, May 11, 2012 at 9:47 AM, Gary Tully <
> gary.tully@gmail.com>
> >> >> wrote:
> >> >> >>>
> >> >> >>>> Do you have a good reason to have networkTTL="5", if there is
> only
> >> one
> >> >> >>>> broker, the default value of 1 will be fine.
> >> >> >>>>
> >> >> >>>> On 11 May 2012 16:32, Chris Robison <chrisdrobison@gmail.com>
> >> wrote:
> >> >> >>>> > If it helps, I've attached the configuration files for both
> >> >> machines.
> >> >> >>>> >
> >> >> >>>> >
> >> >> >>>> > On Fri, May 11, 2012 at 9:07 AM, Chris Robison <
> >> >> >>>> chrisdrobison@gmail.com>
> >> >> >>>> > wrote:
> >> >> >>>> >>
> >> >> >>>> >> One thing I have noticed though is that when I restart the
> >> broker
> >> >> on
> >> >> >>>> the
> >> >> >>>> >> other end, everything starts to work again until it floods
> >> again.
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >> On Fri, May 11, 2012 at 9:05 AM, Chris Robison <
> >> >> >>>> chrisdrobison@gmail.com>
> >> >> >>>> >> wrote:
> >> >> >>>> >>>
> >> >> >>>> >>> I'm on 5.6. And all brokers remain up and running.
> >> >> >>>> >>>
> >> >> >>>> >>>
> >> >> >>>> >>> On Fri, May 11, 2012 at 8:48 AM, Gary Tully <
> >> gary.tully@gmail.com
> >> >> >
> >> >> >>>> wrote:
> >> >> >>>> >>>>
> >> >> >>>> >>>> what version are you on?
> >> >> >>>> >>>> Is there any chance that the broker at the other end of the
> >> >> network
> >> >> >>>> >>>> bridge is shutting down?
> >> >> >>>> >>>>
> >> >> >>>> >>>> On 11 May 2012 14:34, Chris Robison <
> chrisdrobison@gmail.com>
> >> >> >>>> wrote:
> >> >> >>>> >>>> > I have a network of brokers and I am using a
> request-reply
> >> >> system
> >> >> >>>> >>>> > across
> >> >> >>>> >>>> > that network, but I've noticed that after a few
> >> request-replys,
> >> >> >>>> the
> >> >> >>>> >>>> > broker
> >> >> >>>> >>>> > network will get flooded with TempQueue advisory
> messages,
> >> then
> >> >> >>>> some
> >> >> >>>> >>>> > of the
> >> >> >>>> >>>> > of the brokers will be reported as slow consumers and the
> >> >> >>>> consumers
> >> >> >>>> >>>> > won't
> >> >> >>>> >>>> > receive messages after that. The advisory messages seem
> to
> >> be
> >> >> >>>> repeated
> >> >> >>>> >>>> > (e.g. add temp destination, remove temp destination) for
> the
> >> >> same
> >> >> >>>> temp
> >> >> >>>> >>>> > destinations. Not quite sure how to handle this. Any
> >> >> suggestions?
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > Here are some of the errors are other things I'm seeing
> >> >> >>>> repeatedly in
> >> >> >>>> >>>> > logs
> >> >> >>>> >>>> > when I enable debug. In fact, when this flood occurs,
> tons
> >> of
> >> >> log
> >> >> >>>> gets
> >> >> >>>> >>>> > generated all at once.
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 05:24:37,248 | DEBUG | Error occured while
> >> >> processing
> >> >> >>>> async
> >> >> >>>> >>>> > command: DestinationInfo {commandId = 40578,
> >> responseRequired =
> >> >> >>>> false,
> >> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2,
> >> >> destination =
> >> >> >>>> >>>> >
> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
> >> >> >>>> >>>> > operationType =
> >> >> >>>> >>>> > 0, timeout = 0, brokerPath =
> >> >> >>>> [ID:MSSD5D101-38580-1336708634042-0:1]},
> >> >> >>>> >>>> > exception: java.lang.IllegalStateException: Timer already
> >> >> >>>> cancelled. |
> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
> >> >> ActiveMQ
> >> >> >>>> >>>> > Transport: ssl:///10.60.252.28:54493
> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled.
> >> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397)
> >> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
> >> >> >>>> >>>> > at
> >> >> org.apache.activemq.broker.region.Queue.start(Queue.java:889)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 05:24:37,248 | WARN  | Async error occurred:
> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already
> cancelled. |
> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
> >> >> ActiveMQ
> >> >> >>>> >>>> > Transport: ssl:///10.60.252.28:54493
> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled.
> >> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397)
> >> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
> >> >> >>>> >>>> > at
> >> >> org.apache.activemq.broker.region.Queue.start(Queue.java:889)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 05:24:37,249 | DEBUG | mstmip102.vosh.netadding
> >> >> >>>> >>>> > destination:
> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.Connection |
> >> >> >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion |
> >> >> StopAsync:vm://
> >> >> >>>> >>>> > mstmip102.vosh.net#0-23
> >> >> >>>> >>>> > 2012-05-11 05:24:37,250 | WARN  | Failed to remove
> >> connection
> >> >> >>>> >>>> > ConnectionInfo {commandId = 3, responseRequired = false,
> >> >> >>>> connectionId
> >> >> >>>> >>>> > =
> >> >> >>>> >>>> > ID:msipap102-50908-1336690011745-7:1, clientId =
> >> >> >>>> >>>> > mstmip102_msipap102.vosh.net_outbound, clientIp =
> >> >> >>>> >>>> > vm://mstmip102.vosh.net#0,
> >> >> >>>> >>>> > userName = mqipauser, password = *****, brokerPath =
> null,
> >> >> >>>> >>>> > brokerMasterConnector = false, manageable = false,
> >> >> clientMaster =
> >> >> >>>> >>>> > true,
> >> >> >>>> >>>> > faultTolerant = false, failoverReconnect = false},
> reason:
> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already
> cancelled. |
> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
> >> >> >>>> >>>> > StopAsync:vm://
> >> >> >>>> >>>> > mstmip102.vosh.net#0-23
> >> >> >>>> >>>> > 2012-05-11 05:24:37,250 | DEBUG | Exception detail: |
> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service |
> >> >> >>>> >>>> > StopAsync:vm://
> >> >> >>>> >>>> > mstmip102.vosh.net#0-23
> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled.
> >> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397)
> >> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
> >> >> >>>> >>>> > at
> >> >> org.apache.activemq.broker.region.Topic.start(Topic.java:530)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:286)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:481)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:533)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:464)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:459)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConnection(AdvisoryBroker.java:234)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:351)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.security.JaasAuthenticationBroker.removeConnection(JaasAuthenticationBroker.java:95)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:119)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:766)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068)
> >> >> >>>> >>>> > at
> >> >> >>>> >>>> >
> >> >> >>>> >>>> >
> >> >> >>>>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996)
> >> >> >>>> >>>> > at java.lang.Thread.run(Thread.java:722)
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 05:24:37,252 | DEBUG | mstmip102.vosh.netadding
> >> >> >>>> >>>> > destination:
> >> >> >>>> >>>> >
> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1 |
> >> >> >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion |
> ActiveMQ
> >> >> >>>> Transport:
> >> >> >>>> >>>> > ssl:///172.17.5.78:59658
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging
> >> (mstmip102.vosh.net->
> >> >> >>>> >>>> > msipap101.vosh.net)
> >> >> >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184,
> >> >> >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1,
> >> destination
> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath:
> >> >> >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message:
> >> >> ActiveMQMessage
> >> >> >>>> >>>> > {commandId
> >> >> >>>> >>>> > = 0, responseRequired = false, messageId =
> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184,
> >> >> >>>> originalDestination =
> >> >> >>>> >>>> > null,
> >> >> >>>> >>>> > originalTransactionId = null, producerId =
> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination =
> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId =
> null,
> >> >> >>>> expiration
> >> >> >>>> >>>> > = 0,
> >> >> >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588,
> >> >> >>>> >>>> > brokerOutTime =
> >> >> >>>> >>>> > 1336713877341, correlationId = null, replyTo = null,
> >> >> persistent =
> >> >> >>>> >>>> > false,
> >> >> >>>> >>>> > type = Advisory, priority = 0, groupID = null,
> >> groupSequence =
> >> >> 0,
> >> >> >>>> >>>> > targetConsumerId = null, compressed = false, userID =
> null,
> >> >> >>>> content =
> >> >> >>>> >>>> > null,
> >> >> >>>> >>>> > marshalledProperties =
> >> >> >>>> org.apache.activemq.util.ByteSequence@567edec8,
> >> >> >>>> >>>> > dataStructure = DestinationInfo {commandId = 0,
> >> >> responseRequired =
> >> >> >>>> >>>> > false,
> >> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2,
> >> >> destination =
> >> >> >>>> >>>> >
> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
> >> >> >>>> >>>> > operationType =
> >> >> >>>> >>>> > 0, timeout = 0, brokerPath = null}, redeliveryCounter =
> 0,
> >> >> size =
> >> >> >>>> >>>> > 1024,
> >> >> >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616,
> >> >> >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1,
> >> >> >>>> originBrokerName=
> >> >> >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false,
> >> readOnlyBody
> >> >> =
> >> >> >>>> false,
> >> >> >>>> >>>> > droppable = false} |
> >> >> >>>> >>>> >
> org.apache.activemq.network.DemandForwardingBridgeSupport |
> >> >> >>>> ActiveMQ
> >> >> >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging
> >> (mstmip102.vosh.net->
> >> >> >>>> >>>> > msipap101.vosh.net)
> >> >> >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185,
> >> >> >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1,
> >> destination
> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath:
> >> >> >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message:
> >> >> ActiveMQMessage
> >> >> >>>> >>>> > {commandId
> >> >> >>>> >>>> > = 0, responseRequired = false, messageId =
> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185,
> >> >> >>>> originalDestination =
> >> >> >>>> >>>> > null,
> >> >> >>>> >>>> > originalTransactionId = null, producerId =
> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination =
> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId =
> null,
> >> >> >>>> expiration
> >> >> >>>> >>>> > = 0,
> >> >> >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588,
> >> >> >>>> >>>> > brokerOutTime =
> >> >> >>>> >>>> > 1336713877341, correlationId = null, replyTo = null,
> >> >> persistent =
> >> >> >>>> >>>> > false,
> >> >> >>>> >>>> > type = Advisory, priority = 0, groupID = null,
> >> groupSequence =
> >> >> 0,
> >> >> >>>> >>>> > targetConsumerId = null, compressed = false, userID =
> null,
> >> >> >>>> content =
> >> >> >>>> >>>> > null,
> >> >> >>>> >>>> > marshalledProperties =
> >> >> >>>> org.apache.activemq.util.ByteSequence@235448d5,
> >> >> >>>> >>>> > dataStructure = DestinationInfo {commandId = 0,
> >> >> responseRequired =
> >> >> >>>> >>>> > false,
> >> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2,
> >> >> destination =
> >> >> >>>> >>>> >
> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
> >> >> >>>> >>>> > operationType =
> >> >> >>>> >>>> > 1, timeout = 0, brokerPath = null}, redeliveryCounter =
> 0,
> >> >> size =
> >> >> >>>> >>>> > 1024,
> >> >> >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616,
> >> >> >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1,
> >> >> >>>> originBrokerName=
> >> >> >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false,
> >> readOnlyBody
> >> >> =
> >> >> >>>> false,
> >> >> >>>> >>>> > droppable = false} |
> >> >> >>>> >>>> >
> org.apache.activemq.network.DemandForwardingBridgeSupport |
> >> >> >>>> ActiveMQ
> >> >> >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 13:29:40,053 | WARN  | TopicSubscription:
> >> >> >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-4:2:1:1,
> >> >> destinations=7,
> >> >> >>>> >>>> > dispatched=1000, delivered=7723, matched=1001,
> discarded=0:
> >> has
> >> >> >>>> twice
> >> >> >>>> >>>> > its
> >> >> >>>> >>>> > prefetch limit pending, without an ack; it appears to be
> >> slow |
> >> >> >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription |
> >> ActiveMQ
> >> >> >>>> >>>> > Transport:
> >> >> >>>> >>>> > ssl:///10.60.252.28:54614
> >> >> >>>> >>>> >
> >> >> >>>> >>>> > 2012-05-11 13:29:42,540 | WARN  | TopicSubscription:
> >> >> >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-3:2:1:1,
> >> >> destinations=6,
> >> >> >>>> >>>> > dispatched=1000, delivered=9922, matched=1001,
> discarded=0:
> >> has
> >> >> >>>> twice
> >> >> >>>> >>>> > its
> >> >> >>>> >>>> > prefetch limit pending, without an ack; it appears to be
> >> slow |
> >> >> >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription |
> >> ActiveMQ
> >> >> >>>> >>>> > Transport:
> >> >> >>>> >>>> > ssl:///10.60.252.28:54614
> >> >> >>>> >>>>
> >> >> >>>> >>>>
> >> >> >>>> >>>>
> >> >> >>>> >>>> --
> >> >> >>>> >>>> http://fusesource.com
> >> >> >>>> >>>> http://blog.garytully.com
> >> >> >>>> >>>
> >> >> >>>> >>>
> >> >> >>>> >>
> >> >> >>>> >
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> http://fusesource.com
> >> >> >>>> http://blog.garytully.com
> >> >> >>>>
> >> >> >>>
> >> >> >>>
> >> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> http://fusesource.com
> >> >> http://blog.garytully.com
> >> >>
> >>
> >>
> >>
> >> --
> >> http://fusesource.com
> >> http://blog.garytully.com
> >>
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message