activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dejan Bosanac <de...@nighttale.net>
Subject Re: "too many open files" error with 5.3 and Stomp
Date Thu, 22 Oct 2009 13:38:20 GMT
Hi guys,

did some initial testing on this (will commit my test soon, so we can have
it for further testings).

I wasn't able to reproduce the exact error you notice, but was getting a
"unable to create new native thread" from a stomp broker under load. The
test simulates the behavior where producer/consumer open/close conenctions
on each operation.

What I noticed is a lot of sockets in TIME_WAIT (and some in CLOSE_WAIT)
state. After some digging I found that some configuration settings can
improve things

1. Use closeAsync transport option

<transportConnector name="stomp" uri="stomp://
0.0.0.0:61612?transport.closeAsync=false"/>

2. Remove small memoryLimit from destinationPolicy

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true">
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

Since all messages are sent through new connections they are all considered
to be new producers to the broker, and once the flow control is hit,
producers are grouping and probably taking all available connections. You
might try turning off producer flow control completely to see if it works
for you. Found this tip in http://issues.apache.org/activemq/browse/AMQ-1739

3. You can also try some of the scaling techniques, described here
http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html
I turned off dedicated task runner and gave the broker some more memory,
like this
ACTIVEMQ_OPTS="-Xmx1024M -Dorg.apache.activemq.UseDedicatedTaskRunner=false
-Djava.util.logging.config.file=logging.properties"

With all these options, the load test over the standard stomp connector ran
well for quite some time (I still observed some problems with stomp+nio
protocol).

Since there are a few issues regarding stomp and connections

http://issues.apache.org/activemq/browse/AMQ-1739
http://issues.apache.org/activemq/browse/AMQ-1873
https://issues.apache.org/activemq/browse/AMQ-2440

will take more look into it and perhaps when we have it sorted, we should
create activemq-stomp.xml tuned configuration for Stomp usage.

In the meantime, it would be great if you'd test this configuration in your
environment and report if there were any improvements (sample config is
attached)


Cheers
--
Dejan Bosanac - http://twitter.com/dejanb

Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Thu, Oct 22, 2009 at 12:11 PM, DavidLevy <dvid.levy@gmail.com> wrote:

>
> Hi
>
> - it's web, so I can't really tell ...
> I would say 50-100 STOMP PHP producers/consumers
> + 2 REST python producers/consumers
> In 5.3 i was using stomp at first, and then stomp+nio but I think both had
> the issue.
>
> - most of them yes, because it's new PHP scripts
>
> PS : the server is a 64bit server. It might be a problem ?
>
>
>
> Dejan Bosanac wrote:
> >
> > Hi David,
> >
> > ok, can you at least provide more data on:
> >
> > - how many producers/consumers are you using
> > - are you opening a new connection for every message send/receive
> >
> > Also, are you using regular stomp or stomp+nio transports?
> >
> > I'm working on creating a java test that can simulate stomp load
> > environment
> > with many producers/consumers opening connections.
> >
> > Cheers
> > --
> > Dejan Bosanac - http://twitter.com/dejanb
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
> >
> > On Wed, Oct 21, 2009 at 4:03 PM, DavidLevy <dvid.levy@gmail.com> wrote:
> >
> >>
> >> Thanks Dejan,
> >>
> >> This happen in my production environment which is made of a dozen
> server.
> >> I don't believe it would be easy to reproduce by myself in a smaller
> >> environment.
> >>
> >> I noticed that it works better with v5.2. With 5.3, it's not stable for
> >> more
> >> than a few hours. 5.2 can last a few days before it crashes.
> >>
> >> I was also thinking about using REST calls instead of STOMP for some PHP
> >> scripts ... It should help if the STOMP leak is my issue for real ...
> >>
> >>
> >>
> >>
> >>
> >> Dejan Bosanac wrote:
> >> >
> >> > I plan to look at these connection closing issues with stomp in coming
> >> > days.
> >> > Any chance you can create a reproducible test case (in any language)?
> >> >
> >> > Cheers
> >> > --
> >> > Dejan Bosanac - http://twitter.com/dejanb
> >> >
> >> > Open Source Integration - http://fusesource.com/
> >> > ActiveMQ in Action - http://www.manning.com/snyder/
> >> > Blog - http://www.nighttale.net
> >> >
> >> >
> >> > On Wed, Oct 14, 2009 at 12:46 PM, DavidLevy <dvid.levy@gmail.com>
> >> wrote:
> >> >
> >> >>
> >> >> Hi
> >> >>
> >> >> I had the issue with 5.2 and now that I installed 5.3 last night, it
> >> >> still
> >> >> the same, or even worse.
> >> >> It basically looks like this issue :
> >> >> http://issues.apache.org/activemq/browse/AMQ-1873
> >> >>
> >> >> After a short time (1 hour or so) the broker is filled with errors
> >> like
> >> >> 'too
> >> >> many open files' and I have to restart activemq.
> >> >>
> >> >>
> >> >> I don't believe it's a performance issue because
> >> >> - the host server is strong
> >> >> - there are less than 10 queues
> >> >> - there are less than 10000 messages
> >> >>
> >> >> I assume it has to do also with Stomp connections not being released.
> >> >> I am using both PHP and Python clients. I believe the issue is
> related
> >> to
> >> >> PHP clients that may be more error prone if the Apache threads die
> >> >> improperly ...
> >> >>
> >> >> I ll attach my current config file.
> >> >>
> >> >> Any help is welcome !
> >> >>
> >> >> Thanks :)
> >> >>
> >> >> http://www.nabble.com/file/p25888831/activemq.xml activemq.xml
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p25888831.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > -----
> >> > Dejan Bosanac
> >> >
> >> > Open Source Integration - http://fusesource.com/
> >> > ActiveMQ in Action - http://www.manning.com/snyder/
> >> > Blog - http://www.nighttale.net
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p25993154.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > -----
> > Dejan Bosanac
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
>
> --
> View this message in context:
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p26007460.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

Mime
View raw message