Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <14257250.post@talk.nabble.com>
Date: Mon, 10 Dec 2007 09:27:30 -0800 (PST)
From: Hellweek <bwinslow@tfutures.com>
To: users@activemq.apache.org
Subject: Re: (AMQ-CPP) Client excepction when sending (a lot of) messages
In-Reply-To: <14179024.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <13876857.post@talk.nabble.com> <14179024.post@talk.nabble.com>


Having alot of issues with getting a Testcase that will demo the issue that I
am seeing.  What I have found in my testing is a slow consumer can take down
the producers.

To be honest I have spent close to a week on this issue and can not spend
much more time on this.  I will provide my 2 Test Classes in the hopes that
someone can use them to uncover the issue, I will also spend the rest of
today to try and give you folks a clear understanding for whats wrong
however for me to many issues to be able to place this into a production
enviroment.

>From my testing on this issue I can tell you the following.

1. A slow consumer can break the producer.
2. A slow consumer can break the broker.
3. After about 500 active connections the broker becomes very very slow.
4. TTL does not work.
5. Tons of memory issues if you persist messages very very easy to crash the
broker.
6. This issue with the CPP client is and was the last straw for myself and
my team I understand using open source is a risky adventure.
7. Mix Match of version numbers and the fact it  is not clear what Clients
will work with what brokers.


As for me I think ActiveMQ is not ready for proudction use due to the many
issues it has with stability and reliability.  I personaly will be
recomending that we get SonicMQ it is stable and does not require me to
spend weeks figuring out issues that are not releated to my code.  


Hellweek wrote:
> 
> We are alos experiancing the exact same issue.  It seems related to large
> number of producers sending over one connection/session.   I can verify
> this behaviour is the same in the 4.1.1 broker and the 5.0 broker.  It
> does not mater what CPP client is used.  2.1 or 2.1.1 or 2.1.2.
> 
> We have broken down the program to test one producer sending and 2
> producers sending.  It appears the problem is when you have more then 5
> producers sending.
> 
> 
> pfid wrote:
>> 
>> I'm having problems with my client application, not beeing able to
>> send/receive messages (queue blocked) after a certain amount of messages
>> have been send (DeliveryMode::PERSISTENT, Session::CLIENT_ACKNOWLEDGE -
>> all messages get acknowledged upon receiving). Also, there seems to be
>> problems with multiple producer threads (8+) spamming one broker. At 8-9
>> producers writing (different queues) on the same broker at the same time,
>> i get the below mentioned exception.
>> 
>> to reproduce this behaviour i used the example code
>> (http://activemq.apache.org/cms/example.html), and used it to read one of
>> my (full, blocked, exception-throwing-)queues. the program will exit
>> without getting to read anything, with the exception:
>> 
>> 
>> 
>>> No valid response received for command: Begin Class =
>>> ActiveMQTextMessage
>>> Begin Class = ActiveMQMessageBase
>>>  Value of ackHandler = 0
>>>  Value of redeliveryCount = 0
>>>  Value of properties = Begin Class PrimitiveMap:
>>> map[Integer] = 9
>>> Begin Class PrimitiveMap:
>>> 
>>> Begin Class = Message
>>>  Value of Message::ID_MESSAGE = 0
>>>  Value of ProducerId is Below:
>>> Begin Class = ProducerId
>>>  Value of ProducerId::ID_PRODUCERID = 123
>>>  Value of ConnectionId = e41c6cc2-2542-4577-bde0-7011ffd6ea5f
>>>  Value of Value = 0
>>>  Value of SessionId = 0
>>> No Data for Class BaseDataStructure
>>> End Class = ProducerId
>>> 
>>>  Value of Destination is Below:
>>> Begin Class = ActiveMQQueue
>>> Begin Class = ActiveMQDestination
>>>  Value of exclusive = false
>>>  Value of ordered = false
>>>  Value of advisory = false
>>>  Value of orderedTarget = coordinator
>>>  Value of physicalName = TEST.FOO
>>>  Value of options = Begin Class activemq::util::Properties:
>>> End Class activemq::util::Properties:
>>> 
>>> No Data for Class BaseDataStructure
>>> End Class = ActiveMQDestination
>>> End Class = ActiveMQQueue
>>> 
>>>  Value of TransactionId is Below:
>>>    Object is NULL
>>>  Value of OriginalDestination is Below:
>>>    Object is NULL
>>>  Value of MessageId is Below:
>>> Begin Class = MessageId
>>>  Value of MessageId::ID_MESSAGEID = 110
>>>  Value of ProducerId is Below:
>>> Begin Class = ProducerId
>>>  Value of ProducerId::ID_PRODUCERID = 123
>>>  Value of ConnectionId = e41c6cc2-2542-4577-bde0-7011ffd6ea5f
>>>  Value of Value = 0
>>>  Value of SessionId = 0
>>> No Data for Class BaseDataStructure
>>> End Class = ProducerId
>>> 
>>>  Value of ProducerSequenceId = 9
>>>  Value of BrokerSequenceId = 0
>>> No Data for Class BaseDataStructure
>>> End Class = MessageId
>>> 
>>>  Value of OriginalTransactionId is Below:
>>>    Object is NULL
>>>  Value of GroupID =
>>>  Value of GroupSequence = 0
>>>  Value of CorrelationId =
>>>  Value of Persistent = 0
>>>  Value of Expiration = 0
>>>  Value of Priority = 4
>>>  Value of ReplyTo is Below:
>>>    Object is NULL
>>>  Value of Timestamp = 1195642676866
>>>  Value of Type =
>>>  Value of Content[0] = , check broker.
>>>         FILE: activemq/transport/filters/ResponseCorrelator.cpp, LINE:
>>> 146
>>>         FILE: activemq/transport/filters/ResponseCorrelator.cpp, LINE:
>>> 154
>>>         FILE: activemq/connector/openwire/OpenWireFormatNegotiator.cpp,
>>> LINE: 105
>>>         FILE: activemq/connector/openwire/OpenWireConnector.cpp, LINE:
>>> 1370
>>>         FILE: activemq/connector/openwire/OpenWireConnector.cpp, LINE:
>>> 847
>>>         FILE: activemq/core/ActiveMQSession.cpp, LINE: 675
>>>         FILE: activemq/core/ActiveMQProducer.cpp, LINE: 194
>>>         FILE: activemq/core/ActiveMQProducer.cpp, LINE: 149
>>>         FILE: activemq/core/ActiveMQProducer.cpp, LINE: 108
>> 
>> The example code wont be able to read anything from that queue. 
>> 
>> To find out whats causing all this, i re-downloaded the 4.1.1 broker, and
>> ran it using the default activemq.xml file, along with a clean compile of
>> the example code with the following modifications:
>> 
>> 
>>     bool useTopics = false;
>>> 
>>>     HelloWorldProducer producer( brokerURI, 20000, useTopics );
>>>     HelloWorldConsumer consumer( brokerURI, 12000, useTopics );
>>> 
>>>     // Start the consumer thread.
>>> 
>>> //      Thread consumerThread( &consumer );
>>> //      consumerThread.start();
>>> 
>>>     // Wait for the consumer to indicate that its ready to go.
>>> 
>>> //    consumer.waitUnitlReady();
>>> 
>>>     // Start the producer thread.
>>>     Thread producerThread( &producer );
>>>     producerThread.start();
>>> 
>>>     // Wait for the threads to complete.
>>>     producerThread.join();
>>> //    consumerThread.join();
>> 
>> - only write messages
>> - write 20000 at once instead of 2000
>> - dont use topics, use queue instead
>> - no other changes made to example code
>> 
>> On the first run the program will write 20000 messages, increasing the
>> brokers ram usage from initial 40 to 58, and i wont get any errors so
>> far. When starting the program again, it will start writing another 20000
>> messages. Once the brokers ram usage reached almost 70mb, the programm
>> will stop, throwing the above mentioned exception.
>> 
>> Sent message #9291 from thread -1221051472 --> stop here <--
>> 
>> Any further tries starting the program result in the exception before the
>> first message gets send. i can now start receiving the messages, using
>> the same sample code, and use consumer instead of producer. the programm
>> will fetch all messages withouth problems. the ram usage will be still
>> around 70 now. but since all messages are gone, i can send another 20000,
>> further increasing the brokers ram usage to around 87. the program will
>> again throw the exception when i run it a second time.
>> 
>> when i use topics instead of queues, the programm will run without any
>> problems, and the memory increase is barely noticeable.
>> 
>> the broker is started with -Xmx512M (virtual machine), and i tried 
>> 
>> <policyEntry queue="TEST.FOO" memoryLimit="104857600"/>
>> 
>> which doesnt help. setting
>> 
>> <usageManager id="memory-manager" limit="512 MB"/>
>> 
>> will help a bit, i will get the exception at 175mb broker memory usage
>> now (which is after about 10-15 runs of the sample program).
>> 
>> can someone help me with this? with my actual client application i will
>> have 16 (write mode-)queues on the same (4.1.1-)broker, and another
>> client application with 16 (read mode-) queues on the same broker. at the
>> current situation, my "producer"- application will start writing all 16
>> queues upon starting, then exception-exit after a few seconds.
>> 
>> [edit] forgot to mention i'm using activemq-cpp 2.1.1
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/%28AMQ-CPP%29-Client-excepction-when-sending-%28a-lot-of%29-messages-tp13876857s2354p14257250.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.