activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikas Agarwal <vi...@infoobjects.com>
Subject Re: Load issue with ActiveMQ
Date Fri, 07 Nov 2014 06:16:13 GMT
Thanks for pointer Tim. However, I have disabled inactivity check by setting

wireFormat.maxInactivityDuration=0

because earlier we were facing connection breaking because of inactivity.

However, I am able to debug the issue. This time I looked into one of the
rolled log file and found that there was a few errors saying "too many open
files". This triggered me to think over monitoring current threads and I
found that number of threads crossed the limits of default system limit
(ulimit -a) 1024. We were running a few more processes on the same machine
with the same user which are actually using few hundreds of threads which
caused the limits to reach quickly. So, I increased the system limits for
threads and open files by copying the limit configuration for yarn user
from other machine which increased the values as

<username> - nofile 32768
<username> - nproc 65536

And after setting these values and restarting the machine, I observed the
system for 2-3 days and it worked. System didn't show the crash behaviour I
mentioned.


Thanks everyone for your valuable time and suggestions. :)

On Tue, Nov 4, 2014 at 10:36 PM, Tim Bain <tbain@alumni.duke.edu> wrote:

> Can you try increasing the wire format's maxInactivityDurationInitialDelay
> as described by Christian in a similar-sounding problem last year:
>
> http://activemq.2283324.n4.nabble.com/Wire-Format-Negotiation-Timeout-Say-what-td4666972.html
>
> That may not be the final solution (there might be underlying network
> problems that need to be addressed by your admins), but it should help
> diagnose if that's what's going on...
>
> On Mon, Nov 3, 2014 at 10:17 PM, Vikas Agarwal <vikas@infoobjects.com>
> wrote:
>
> > For more information, I see following messages in logs (whenever,
> ActiveMQ
> > fails to respond to producer and consumer but logs are available)
> >
> > 2014-11-04 00:08:03,893 | WARN  | Transport Connection to:
> tcp://<ip>:45550
> > failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///<ip>:45550
> >
> >
> > And on client side
> >
> > WARN  org.apache.activemq.transport.failover.FailoverTransport  -
> Transport
> > (tcp://<ip>:61616) failed, reason:  java.io.IOException: Wire format
> > negotiation timeout: peer did not send his wire format., attempting to
> > automatically reconnect
> >
> >
> > Strange things is that with this state, everything works sometimes while
> > does not work at rest of the time.
> >
> > On Tue, Nov 4, 2014 at 10:24 AM, Vikas Agarwal <vikas@infoobjects.com>
> > wrote:
> >
> > > By crash, I mean message producers are not able to connect to ActiveMQ
> > and
> > > queue listeners stop processing messages because of the same reason. I
> > have
> > > to restart ActiveMQ to make everything working fine.
> > >
> > > On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <tbain@alumni.duke.edu>
> wrote:
> > >
> > >> Most developers I know would define "crash" to be "the process
> > >> terminated".  Since that doesn't seem to be the case here (and since
> you
> > >> applied the term to an instance of Producer Flow Control in a previous
> > >> situation, which is most definitely not an ActiveMQ crash nor an
> > ActiveMQ
> > >> error of any sort), I assume that's not your definition of the term.
> > Can
> > >> you describe the symptoms you're seeing, without using the word
> "crash",
> > >> to
> > >> get everyone on the same page about what your problem actually is?
> > >>
> > >> On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <vikas@infoobjects.com
> >
> > >> wrote:
> > >>
> > >> > I mean 50,000-100,000 messages are there in queue after which it
> > crashes
> > >> > and this range is not definitive, it some crashes below a pile of
> > 50,000
> > >> > messages too. Actually, I am not able to predict when exactly it is
> > >> > crashing, if its is crashing due to CPU usage or memory usage or IO.
> > >> >
> > >> > Logs are behaving fine before the crash, so I believe everything is
> > fine
> > >> > with permissions and log level. Log rotation would not remove the
> > >> > activemq.log completely. Further, after crash, activemq process is
> > still
> > >> > alive (checked using ps command), however, Admin UI (port 8161) is
> not
> > >> > available after crash.
> > >> >
> > >> > On Mon, Nov 3, 2014 at 7:03 PM, James Green <
> james.mk.green@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> > > On 3 November 2014 09:42, Vikas Agarwal <vikas@infoobjects.com>
> > >> wrote:
> > >> > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > We are using ActiveMQ for 3+ years, however, we didn't test
it
> > under
> > >> > > heavy
> > >> > > > load. Recently, we started using ActiveMQ in another project
> where
> > >> > > > sometimes load increases exponentially as we are listening
to
> > >> twitter
> > >> > > > stream which can pile up a lot of messages depending of
> occurrence
> > >> of
> > >> > > some
> > >> > > > global event like iPhone 6 launch. Now, what we are seeing
in
> > >> ActiveMQ
> > >> > > > admin UI is that one of the queues gets piled up with thousands
> of
> > >> > > message
> > >> > > > ranging from 50k-100k. Now, sometimes (almost daily) the
> ActiveMQ
> > >> > crashes
> > >> > > >
> > >> > >
> > >> > > Do you mean:
> > >> > >
> > >> > > 1. The queues have 50-100,000 messages each, or
> > >> > > 2. The queues have thousands of messages each of which may be
> > 50-100k
> > >> in
> > >> > > size?
> > >> > >
> > >> > >
> > >> > > > in between of my sleeping hours. :( And the strange thing
is
> that
> > >> when
> > >> > > try
> > >> > > > to view the log file for the reason of the crash, I am not
able
> to
> > >> find
> > >> > > the
> > >> > > > activemq.log file at all. We earlier faced ActiveMQ crash
> issues,
> > >> but
> > >> > > that
> > >> > > >
> > >> > >
> > >> > > Ensure you have INFO logging switched on and that AMQ writes
to
> this
> > >> log
> > >> > on
> > >> > > start-up. Also, that log could be rotated - ensure the process
has
> > >> > > permission to write to it.
> > >> > >
> > >> > >
> > >> > > > was due to producer flow control and we disabled that to
fix the
> > >> issue
> > >> > > and
> > >> > > > main point is that in those cases we never had the case
where
> the
> > >> log
> > >> > > file
> > >> > > > was missing. I have to restart the process to fix the issue
and
> > that
> > >> > too,
> > >> > > > twice because on first restart, it shows error about missing
or
> > >> corrupt
> > >> > > > data log.
> > >> > > >
> > >> > > > So, please suggest me where to look for such issue or what
> should
> > >> be my
> > >> > > > next move for debugging. I am stuck with missing log file.
> > >> > > >
> > >> > > > For information here are the memory settings:
> > >> > > >
> > >> > > > <systemUsage>
> > >> > > >     <systemUsage>
> > >> > > >         <memoryUsage>
> > >> > > >             <memoryUsage limit="*256 mb*"/>
> > >> > > >         </memoryUsage>
> > >> > > >         <storeUsage>
> > >> > > >             <storeUsage limit="*100 gb*"/>
> > >> > > >         </storeUsage>
> > >> > > >         <tempUsage>
> > >> > > >             <tempUsage limit="*50 gb*"/>
> > >> > > >         </tempUsage>
> > >> > > >     </systemUsage>
> > >> > > > </systemUsage>
> > >> > > >
> > >> > >
> > >> > > You might want to start here and work out the model of usage
> you're
> > >> > seeing:
> > >> > > http://activemq.apache.org/javalangoutofmemory.html
> > >> > >
> > >> > >  For instance: thousands of messages, few clients; or thousands
of
> > >> > messages
> > >> > > and lots of clients.
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Vikas Agarwal
> > >> > 91 – 9928301411
> > >> >
> > >> > InfoObjects, Inc.
> > >> > Execution Matters
> > >> > http://www.infoobjects.com
> > >> > 2041 Mission College Boulevard, #280
> > >> > Santa Clara, CA 95054
> > >> > +1 (408) 988-2000 Work
> > >> > +1 (408) 716-2726 Fax
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Vikas Agarwal
> > > 91 – 9928301411
> > >
> > > InfoObjects, Inc.
> > > Execution Matters
> > > http://www.infoobjects.com
> > > 2041 Mission College Boulevard, #280
> > > Santa Clara, CA 95054
> > > +1 (408) 988-2000 Work
> > > +1 (408) 716-2726 Fax
> > >
> > >
> >
> >
> > --
> > Regards,
> > Vikas Agarwal
> > 91 – 9928301411
> >
> > InfoObjects, Inc.
> > Execution Matters
> > http://www.infoobjects.com
> > 2041 Mission College Boulevard, #280
> > Santa Clara, CA 95054
> > +1 (408) 988-2000 Work
> > +1 (408) 716-2726 Fax
> >
>



-- 
Regards,
Vikas Agarwal
91 – 9928301411

InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message