activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ajls77 <ajlstu...@gmail.com>
Subject Solved: Messages appear to get lost
Date Fri, 30 Nov 2007 16:03:16 GMT

Hmmmm ... as suspected, a silly error in the client thread model was causing
two threads to use the same producer.  



ajls77 wrote:
> 
> Thanks for the response.  I have bumped by my usageManager's resource
> allocation to 512Mb and the problem still persists.  Funnily enough, it
> seems easier to replicate the problem with 10 threads producing every
> second than 250 per second.  So it cannot be an out and out
> concurrency-induced.  I am still quite suspicious that my thread model
> (ThreadPoolExecutor - and some LockSupport stuff) may be causing the
> problem.
> 
> To verify this, could you help clarify these assumptions ?
> 
> When TcpBufferedOutputStream.flush() is called, and useAsyncSend(false),
> the message, as far as the JVM is concerned, has "left the building".
> 
> and
> 
> If not, what part of ActiveMQ could affect the sending after this point.
> 
> 
> I am convinced that TcpBufferedOutputStream.flush() is executing correctly
> for a given message ID.   
> 
> Many thanks
> 
> P.S. I cannot, as of now use 5.x, but the message cursors will give a nice
> amount of control over dispatching.  
> 
> 
> James.Strachan wrote:
>> 
>> Maybe you're hitting the flow control causing some producers to be
>> blocked. See this FAQ entry...
>> http://activemq.apache.org/my-producer-blocks.html
>> 
>> You might wanna try using 5.x which has much better support for
>> dealing with flow control in different ways.
>> 
>> 
>> On 27/11/2007, ajls77 <ajlstuart@gmail.com> wrote:
>>>
>>> I am sending messages to an ActiveMQ broker from an Java5
>>> ExecutorService
>>> thread pool.  Concurrency requirements at the moment require at least 60
>>> MessageProducers - these, along with a Session, live in a thread-safe
>>> FIFO
>>> pipe.
>>>
>>> When concurrency hots up, it appears that some messages are being lost -
>>> from the client's perspective it appears to have been delivered, but the
>>> debugging output on the broker does not provide a receipt of the message
>>> (the message delivery is set to PERSISTENT just to get the debugging as
>>> there is no debugging information under NON_PERSISTENT).
>>>
>>> To prove that the client thinks the message is delivered I recompiled
>>> activemq-core-4.1.1.jar with extra debugging.  I traced the following
>>> execution flow:
>>>
>>> ActiveMQMessageProducer.send()
>>> -> ActiveMQSession.send()
>>>  -> TcpTransport.oneway()
>>>    -> TcpBufferedOutputStream.flush()
>>>
>>> The output of which is:
>>>
>>> DEBUG: org.apache.activemq.ActiveMQSession - sending message with
>>> messageID:
>>> ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
>>> Thread[pool-1-thread-493,5,main]
>>> DEBUG: org.apache.activemq.ActiveMQSession - Sending message:
>>> ActiveMQTextMessage {commandId = 0, responseRequired = false, messageId
>>> =
>>> ID:ukedfrog-3905-1196171536066-1:5:1:1:54, originalDestination = null,
>>> originalTransactionId = null, producerId =
>>> ID:ukedfrog-3905-1196171536066-1:5:1:1, destination =
>>> queue://urn:telelogic:ers:two, transactionId = null, expiration = 0,
>>> timestamp = 1196171643911, arrival = 0, correlationId = null, replyTo =
>>> queue://reply-192.168.68.151@urn:telelogic:ers:two, persistent = true,
>>> type
>>> = null, priority = 4, groupID = null, groupSequence = 0,
>>> targetConsumerId =
>>> null, compressed = false, userID = null, content = null,
>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>> 0,
>>> size = 0, properties = null, readOnlyProperties = true, readOnlyBody =
>>> true,
>>> droppable = false, text = messageContent}
>>> DEBUG org.apache.activemq.transport.tcp.TcpBufferedOutputStream calling
>>> flush - messageID: ID:ukedfrog-3905-1196171536066-1:5:1:1:54 count: 116
>>> OutputStream: java.net.SocketOutputStream@94124f
>>> DEBUG org.apache.activemq.transport.tcp.TcpTransport flushing bytes -
>>> messageID:ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
>>> Thread[pool-1-thread-493,5,main]
>>>
>>> (The messageID is propagated to the activemq classes via a ThreadLocal)
>>>
>>> My understanding is that after TcpBufferedOutputStream.flush() the TCP
>>> packet is delegated to the underlying operating system and the message
>>> has
>>> effectively been "sent".  However, there is no receipt of this message
>>> in
>>> the broker, no matter how long we wait.  Is this message "lost" ? or is
>>> my
>>> client implementation faulty.
>>>
>>> I am using one Connection per JVM and used by each thread (with its own
>>> MessageProducer & Session). After reading
>>> http://gnodet.blogspot.com/2006/12/over-past-weeks-i-spend-some-times-load.html
>>> this  , I tried using a separate Connection per thread, but the problem
>>> persisted.  This write-up concerns me  (that the broker processes all
>>> messages on a given Connection sequentially)  as this would force the
>>> client
>>> implementation to have multiple Connections which does not seem to be
>>> mandated by the JMS spec.
>>>
>>>
>>> My ConnectionFactory is configured to use synchronous sending via
>>> "connectionFactory.setUseAsyncSend(false);"
>>>
>>> Could anybody explain why these message appear to be "lost"? It seems,
>>> from
>>> client perspective, that the message has been delivered, but it never
>>> is.
>>>
>>> Many thanks
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a13971307
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>> 
>> 
>> -- 
>> James
>> -------
>> http://macstrac.blogspot.com/
>> 
>> Open Source Integration
>> http://open.iona.com
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a14076065
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message