activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Anderson <manderso...@gmail.com>
Subject Re: Memory and Temp Usage Questions
Date Tue, 27 Nov 2012 14:31:36 GMT
PeerNetworkConnector extends DiscoveryNetworkConnector so I can fire
listeners for onServiceAdd and onServiceRemove.


On 27 November 2012 14:16, Christian Posta <christian.posta@gmail.com>wrote:

> Okay good to know. I suppose this error happened just once randomly and you
> cannot reproduce?
>
> BTW... what is PeerNetworkConnector in your config:
>
>     NetworkConnector networkConnector = new
> PeerNetworkConnector(peerAddress, uri, this);
>
>
> On Tue, Nov 27, 2012 at 7:08 AM, Mark Anderson <manderson23@gmail.com
> >wrote:
>
> > The prefetch size was set on the network connector as we were getting
> > messages about slow consumers across the network bridge.
> >
> > As far as I can see the network bridge had not failed. The connector
> > entries in the log are for a client subscription that will also have the
> > topic prefetch set to 32766. I am trying to get logs from the client.
> >
> > The broker on the other end of the bridge uses the same configuration.
> >
> >
> > On 27 November 2012 13:41, Christian Posta <christian.posta@gmail.com
> > >wrote:
> >
> > > Answers to your questions:
> > >
> > > 1) Not sure yet
> > > 2) Because at the moment, send fail if no space is only triggered when
> > > producer flow control is on (at least for this case, topics)
> > > 3) like gtully said, connections could not be shutdown if they are
> > blocked
> > > somehow
> > >
> > > I noticed in your config you explicitly set the prefetch on the network
> > > connector to 32766. The default for network connectors is 1000 and the
> > > default for regular topics is Short.MAX_VALUE (which is 32767). Since
> the
> > > bridge doesn't have a prefetch buffer like normal clients do, setting
> the
> > > prefetch to 32766 could end up flooding it. Any reason why you have it
> > set
> > > to 32766?
> > >
> > > So TopicSubscriptions should always have the broker's main memory
> usage.
> > If
> > > it has the destination's memory limit, then something went wrong. Like
> > Gary
> > > said, the pending message cursor's messages would be spooled to disk
> when
> > > the main memory limit reaches its high water mark (70% by default)....
> > but
> > > that appears to not have happened in this case.
> > >
> > > Are there any indications that the TopicSubscription is for the network
> > > bridge? Or maybe that the network bridge failed somehow? I see that the
> > > dispatched count is the same as what you've set for you prefertch on
> the
> > > bridge, but if anything else can point to that it might be helpful. For
> > > example, are those port numbers on the transport connector logs for the
> > > network bridge?
> > >
> > > How is the broker on the other end of the bridge configured? Same?
> > >
> > >
> > > On Fri, Nov 23, 2012 at 8:36 AM, Mark Anderson <manderson23@gmail.com
> > > >wrote:
> > >
> > > > I have ActiveMQ 5.6.0 configured as follows:
> > > >
> > > > Producer Flow Control = false
> > > > Send Fail If No Space = true
> > > > Memory Usage Limit = 128Mb
> > > > Temp Usage Limit = 1Gb
> > > >
> > > > All my messages are non-persistent. The temp usage is configured to
> > > handle
> > > > spikes/slow consumers when processing messages.
> > > >
> > > > I continually see the following in the logs:
> > > >
> > > > WARN  Nov 20 20:55:47 (13748874 [InactivityMonitor Async Task:
> > > > java.util.concurrent.ThreadPoolExecutor$Worker@7ea0e15b[State = 0,
> > empty
> > > > queue]] org.apac
> > > > he.activemq.broker.TransportConnection.Transport) Transport
> Connection
> > > to:
> > > > tcp://192.168.2.103:35186 failed: java.net.SocketException: Broken
> > pipe
> > > > INFO  Nov 20 20:55:51 (13752162 [ActiveMQ Transport: tcp:///
> > > > 192.168.2.103:35168] org.apache.activemq.broker.TransportConnection)
> > The
> > > > connection to 'tcp:
> > > > //192.168.2.103:35166' is taking a long time to shutdown.
> > > > INFO  Nov 20 20:55:56 (13757162 [ActiveMQ Transport: tcp:///
> > > > 192.168.2.103:35168] org.apache.activemq.broker.TransportConnection)
> > The
> > > > connection to 'tcp:
> > > > //192.168.2.103:35166' is taking a long time to shutdown.
> > > > INFO  Nov 20 20:56:01 (13762162 [ActiveMQ Transport: tcp:///
> > > > 192.168.2.103:35168] org.apache.activemq.broker.TransportConnection)
> > The
> > > > connection to 'tcp:
> > > > //192.168.2.103:35166' is taking a long time to shutdown.
> > > >
> > > > I'm not sure why the connection will never shutdown.
> > > >
> > > > I then see the following message:
> > > >
> > > > org.apache.activemq.broker.region.TopicSubscription)
> TopicSubscription:
> > > > consumer=ID:linux-5ks2-57958-1353426643811-3:1:378:1, destinations=1,
> > > > dispatched=32766, delivered=0, matched=0, discarded=0: Pending
> message
> > > > cursor
> > > >
> > > >
> > >
> >
> [org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@4c41cfa2
> > > > ]
> > > > is full, temp usage (0%) or memory usage (211%) limit reached,
> blocking
> > > > message add() pending the release of resources.
> > > >
> > > > This leads me to the following questions:
> > > >
> > > > 1) Why would the memory usage be 211% while temp usage is 0%.
> > > > 2) The thread dump shows that send calls on producers are blocking.
> Why
> > > > would they not throw exceptions when send fail if no space = true?
> > > > 3) Would the issue with connection shutdown contribute to the memory
> > > usage?
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > http://www.christianposta.com/blog
> > > twitter: @christianposta
> > >
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message