Hi guys,

did some initial testing on this (will commit my test soon, so we can have it for further testings).

I wasn't able to reproduce the exact error you notice, but was getting a "unable to create new native thread" from a stomp broker under load. The test simulates the behavior where producer/consumer open/close conenctions on each operation.

What I noticed is a lot of sockets in TIME_WAIT (and some in CLOSE_WAIT) state. After some digging I found that some configuration settings can improve things

1. Use closeAsync transport option

<transportConnector name="stomp" uri="stomp://0.0.0.0:61612?transport.closeAsync=false"/>

2. Remove small memoryLimit from destinationPolicy

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true">
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

Since all messages are sent through new connections they are all considered to be new producers to the broker, and once the flow control is hit, producers are grouping and probably taking all available connections. You might try turning off producer flow control completely to see if it works for you. Found this tip in http://issues.apache.org/activemq/browse/AMQ-1739

3. You can also try some of the scaling techniques, described here http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html
I turned off dedicated task runner and gave the broker some more memory, like this
ACTIVEMQ_OPTS="-Xmx1024M -Dorg.apache.activemq.UseDedicatedTaskRunner=false -Djava.util.logging.config.file=logging.properties"

With all these options, the load test over the standard stomp connector ran well for quite some time (I still observed some problems with stomp+nio protocol).

Since there are a few issues regarding stomp and connections

http://issues.apache.org/activemq/browse/AMQ-1739
http://issues.apache.org/activemq/browse/AMQ-1873
https://issues.apache.org/activemq/browse/AMQ-2440

will take more look into it and perhaps when we have it sorted, we should create activemq-stomp.xml tuned configuration for Stomp usage.

In the meantime, it would be great if you'd test this configuration in your environment and report if there were any improvements (sample config is attached)


Cheers
--
Dejan Bosanac - http://twitter.com/dejanb

Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Thu, Oct 22, 2009 at 12:11 PM, DavidLevy <dvid.levy@gmail.com> wrote:

Hi

- it's web, so I can't really tell ...
I would say 50-100 STOMP PHP producers/consumers
+ 2 REST python producers/consumers
In 5.3 i was using stomp at first, and then stomp+nio but I think both had
the issue.

- most of them yes, because it's new PHP scripts

PS : the server is a 64bit server. It might be a problem ?



Dejan Bosanac wrote:
>
> Hi David,
>
> ok, can you at least provide more data on:
>
> - how many producers/consumers are you using
> - are you opening a new connection for every message send/receive
>
> Also, are you using regular stomp or stomp+nio transports?
>
> I'm working on creating a java test that can simulate stomp load
> environment
> with many producers/consumers opening connections.
>
> Cheers
> --
> Dejan Bosanac - http://twitter.com/dejanb
>
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>
>
> On Wed, Oct 21, 2009 at 4:03 PM, DavidLevy <dvid.levy@gmail.com> wrote:
>
>>
>> Thanks Dejan,
>>
>> This happen in my production environment which is made of a dozen server.
>> I don't believe it would be easy to reproduce by myself in a smaller
>> environment.
>>
>> I noticed that it works better with v5.2. With 5.3, it's not stable for
>> more
>> than a few hours. 5.2 can last a few days before it crashes.
>>
>> I was also thinking about using REST calls instead of STOMP for some PHP
>> scripts ... It should help if the STOMP leak is my issue for real ...
>>
>>
>>
>>
>>
>> Dejan Bosanac wrote:
>> >
>> > I plan to look at these connection closing issues with stomp in coming
>> > days.
>> > Any chance you can create a reproducible test case (in any language)?
>> >
>> > Cheers
>> > --
>> > Dejan Bosanac - http://twitter.com/dejanb
>> >
>> > Open Source Integration - http://fusesource.com/
>> > ActiveMQ in Action - http://www.manning.com/snyder/
>> > Blog - http://www.nighttale.net
>> >
>> >
>> > On Wed, Oct 14, 2009 at 12:46 PM, DavidLevy <dvid.levy@gmail.com>
>> wrote:
>> >
>> >>
>> >> Hi
>> >>
>> >> I had the issue with 5.2 and now that I installed 5.3 last night, it
>> >> still
>> >> the same, or even worse.
>> >> It basically looks like this issue :
>> >> http://issues.apache.org/activemq/browse/AMQ-1873
>> >>
>> >> After a short time (1 hour or so) the broker is filled with errors
>> like
>> >> 'too
>> >> many open files' and I have to restart activemq.
>> >>
>> >>
>> >> I don't believe it's a performance issue because
>> >> - the host server is strong
>> >> - there are less than 10 queues
>> >> - there are less than 10000 messages
>> >>
>> >> I assume it has to do also with Stomp connections not being released.
>> >> I am using both PHP and Python clients. I believe the issue is related
>> to
>> >> PHP clients that may be more error prone if the Apache threads die
>> >> improperly ...
>> >>
>> >> I ll attach my current config file.
>> >>
>> >> Any help is welcome !
>> >>
>> >> Thanks :)
>> >>
>> >> http://www.nabble.com/file/p25888831/activemq.xml activemq.xml
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p25888831.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > -----
>> > Dejan Bosanac
>> >
>> > Open Source Integration - http://fusesource.com/
>> > ActiveMQ in Action - http://www.manning.com/snyder/
>> > Blog - http://www.nighttale.net
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p25993154.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> -----
> Dejan Bosanac
>
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>

--
View this message in context: http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p26007460.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.