Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of bruce.snyder@gmail.com
 designates 74.125.44.157 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=hnzHX8zCTMbAlSt4zgXFykg8A9bKp4xlMMG4+fBvhY//AVGNOupQBUNND44ZEjaD83
         zSkZt3e4ai3nGcB3rZWA6uETRpj6tltYSJoxxtTWn8EX/0SmCGzia6+BTWROy/rAA8oQ
         T9VlfO8XQbySvXx9UHSqblnqoRN/Aj+2uKQ2Q=
Message-ID: <7b3355cb0808212111y63e11b3i187919f6c847939e@mail.gmail.com>
Date: Thu, 21 Aug 2008 22:11:20 -0600
From: "Bruce Snyder" <bruce.snyder@gmail.com>
To: users@activemq.apache.org, users@servicemix.apache.org
Subject: Re: ActiveMQ causes producer to hang when no consumer on topic after
 a few minutes.
In-Reply-To: <fa74450c0808212043h8094681s41a585ca7ffb377a@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <fa74450c0808191754w2db20629tc3a3f040121aed68@mail.gmail.com>
	 <fa74450c0808191925n3f84edd1jc5116250fdfa8108@mail.gmail.com>
	 <7b3355cb0808201303q5c5f7389o297ef12a63e4ddaa@mail.gmail.com>
	 <fa74450c0808201349l4ddddaf8nf7f4bce8e007a9bb@mail.gmail.com>
	 <7b3355cb0808201405m3e7efa4ft7a33ffd1f05da56@mail.gmail.com>
	 <fa74450c0808202039g5465741v76261b42e66cfb18@mail.gmail.com>
	 <fa74450c0808212043h8094681s41a585ca7ffb377a@mail.gmail.com>

On Thu, Aug 21, 2008 at 9:43 PM, Ryan Moquin <fragility2.0@gmail.com> wrote:
> Hi Bruce,
>
> I tried my service with the two settings:
>
> deliveryMode="1"
> timeToLive="1000"
>
> The scenario is an external JMS client sends a message to a JMS consumer SU
> in servicemix.  It forwards that to my service which does some processing
> with the incoming message, and then creates a new message that it sends to a
> JMS provider SU.  I ran this using Servicemix 3.3.
>
> I loaded up jconsole and watched my provider topic.  As soon as my Topic
> size hit 1321, my service all of a sudden stopped receiving messages from my
> external sender and went completely silent.  If I look at the topic that my
> external sender, is sending to, it is still filling up with messages that
> are no longer being received by my hanging service.  When my external sender
> finished, I see that 1500 messages were enqueued and dequeued on my consumer
> JMS SU, but only 1340 were enqueued on my provider JMS SU.
>
> So, I did a stack dump on the JVM and noticed there are 35 hanging threads
> that are all hanging trying to send a message to the JMS provider.  So
> apparently, the:
>
> <property name="useAsyncSend" value="true" />
>
> is working and I just didn't notice that there were 35 hanging threads.  The
> problem is, I guess once enough threads are tied up because the messages
> can't be sent, then everything starts hanging.  I have the message time to
> live sent to a 1000.  You said that it worked in 5.1, but it doesn't seem to
> be working for me.  Is that value really in milliseconds?  Is there
> something else I need?  The non persistent delivery mode obviously isn't
> working either.  Do I have any options for preventing this situation from
> happening?  I don't see any other options in the config I can try.
>
> Here is the stack dump that is the same for all 35 threads:
>
> "pool-component.servicemix-jms-thread-3" prio=6 tid=0x39a4b400 nid=0x2724
> waiting on condition [0x3f58e000..0x3f58fb14]
>   java.lang.Thread.State: WAITING (parking)
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for  <0x1774ee48> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:19

I'm thinking that the issues you're experiencing are ServiceMix
related, not ActiveMQ related. It sounds like messages may not be
consumed fast enough which could cause the slowdown you describe, but
I'm not sure yet. I'd like to see how you have the thread pooling
configured in ServiceMix, so please post your servicemix.properties
file. Also, please provide two full thread dumps (kill -3 <pid>) of
ServiceMix that are, maybe say, 20-30 seconds apart. These three items
should help us to troubleshoot the situation a bit further.

Bruce
-- 
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'

Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/

Blog: http://bruceblog.org/