activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Langford <danlangf...@gmail.com>
Subject Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Date Wed, 22 Aug 2018 23:28:28 GMT
i validated that the fix did in fact fix our upgrade path. i made a new
2.1.0 broker. i added some address settings via JMX. i attempted an upgrade
to 2.6.2 and watched it fail. i pulled the code that you linked above. i
packaged a release and then i upgraded to the released code and the server
started just fine.

thank you so much!!

what is the timeline on a 2.6.3 or 2.7.0 release?

On Fri, Aug 3, 2018 at 4:22 PM Clebert Suconic <clebert.suconic@gmail.com>
wrote:

> I'm considering doing a 2.6.3 very soon.. it will include the fix..
> you should be able to move from 2.1.0 . let me know if you have any
> other issues.
>
> On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic
> <clebert.suconic@gmail.com> wrote:
> > https://github.com/apache/activemq-artemis/pull/2214
> >
> >
> > I will make it into 2.6.x as soon as the PR tests are finished.
> >
> >
> > If you could try a snapshot of master or 2.6.x (after I merge it of
> > course). Let me know if you need help buildling it.
> >
> > On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford <danlangford@gmail.com>
> wrote:
> >> Michael for the win!
> >> @Clebert im not opposed to coding up a test case it is just that i am
> >> swamped at the moment so im glad to hear that Michael already has a
> bead on
> >> this
> >>
> >> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
> >> imp* on a clean 2.6.2. the new system didnt have Address Settings and
> >> Security Settings that had previously been added via the jolokia api. i
> >> should look into the export options more to see if i just failed to
> provide
> >> a command line argument. fortunately in my scenario it was pretty
> trivial
> >> to recreate those settings quickly. so i did that. the import created
> all
> >> the addresses, routes, queues, and messages that i needed. it went
> pretty
> >> well. so my cluster is now on 2.6.2
> >>
> >> i plan to rework our usage of AddressSetting to not create a new entry
> for
> >> every Address we create but instead only create them for the few
> exceptions
> >> to our default configuration
> >>
> >> then i can start working on running the broker in CF with the cluster in
> >> more of a Master-Master mode.
> >> thanks again for all your help. feel free to respond if there is any
> more
> >> data you need from me.
> >>
> >> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
> >> while handshaking has occurred. " but thats another issue i will figure
> >> out. *
> >>
> >> my 30 minute startup times are now down to 30 seconds max. more like
> 10-15
> >> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.
> >>
> >> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
> >> <michael.andre.pearce@me.com.invalid> wrote:
> >>
> >>> Clebert im able to recreate this and also worked out in code why this
> >>> happens. Ill try catch you on irc tomorrow or friday.
> >>>
> >>> Sent from my Samsung Galaxy smartphone.
> >>> -------- Original message --------From: Clebert Suconic <
> >>> clebert.suconic@gmail.com> Date: 25/07/2018  20:20  (GMT+00:00) To:
> >>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+
> minutes
> >>> to boot & failover
> >>> Yes.. it should include it.
> >>>
> >>>
> >>> Do you have an example of the API you used with 2.1?
> >>>
> >>>
> >>> I can't write a compatibility test this week.. but if you provide me
> >>> the example I will do it early next week.
> >>>
> >>>
> >>> In a perfect world, if you did it .. it would be great :)
> >>>
> >>>
> >>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <danlangford@gmail.com>
> >>> wrote:
> >>> > i tried 2.6.2 this morning to see if that was an improvement from
> 2.5.0 i
> >>> > tried a few months ago. unfortunately there was not. it may have
> failed
> >>> > much faster than 2.5.0 i dont recall the timing specifics but the
> error
> >>> is
> >>> > the same
> >>> >
> >>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server]
> AMQ224000:
> >>> > Failure in initialisation: java.lang.NegativeArraySizeException
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > [rt.jar:1.8.0_171]
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>> > [rt.jar:1.8.0_171]
> >>> > at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > [rt.jar:1.8.0_171]
> >>> > at java.lang.reflect.Method.invoke(Method.java:498)
> [rt.jar:1.8.0_171]
> >>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>> > [artemis-boot.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>> > [artemis-boot.jar:2.6.2]
> >>> >
> >>> >
> >>> > right there in the AddressSettings.decode
> >>> >
> >>> >
> >>> > I am going to start looking into a journal export and then a journal
> >>> import
> >>> > and see what happens there.
> >>> > are programmatically created addresses / queues stored in the data
> export
> >>> > along with the address and security settings?
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
> >>> clebert.suconic@gmail.com>
> >>> > wrote:
> >>> >
> >>> >> @dan take a look at the compatibility tests in master.. We're help
> to
> >>> >> help if you don't understand anything.. it's using ClassLoaders
and
> >>> >> Groovy so you can produce stuff with a combination of clients and
> >>> >> servers on 2.1.0 and consume on the current version.
> >>> >>
> >>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
> >>> >> <clebert.suconic@gmail.com> wrote:
> >>> >> > @Dan: what about this:
> >>> >> >
> >>> >> > you provide us a code path that could generate the
> incompatibility in
> >>> >> 2.1.0...
> >>> >> >
> >>> >> > an ultimate deal would be if you produced a compatibility
test
> with
> >>> >> > 2.1.0.... you add some code that will generate the journal
with
> 2.1.0,
> >>> >> > and then consume the messages on master (or 2.6.x)...
> >>> >> >
> >>> >> >
> >>> >> > if you generate the compatibility test showing the issue,
we can
> try
> >>> >> fixing it.
> >>> >> >
> >>> >> >
> >>> >> > What about this idea? that would help you migrate into 2.6.x
or
> >>> master.
> >>> >> >
> >>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
> >>> >> > <clebert.suconic@gmail.com> wrote:
> >>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <
> danlangford@gmail.com
> >>> >
> >>> >> wrote:
> >>> >> >>> Thank you that was very helpful. we actually do have
an address
> >>> >> settings
> >>> >> >>> entry for each queue. there could be a better pattern
for us.
> but
> >>> >> currently
> >>> >> >>> our automated system for creating queues creates an
address
> setting
> >>> at
> >>> >> the
> >>> >> >>> same time. i will look into improved patterns.
> >>> >> >>>
> >>> >> >>> as far as upgrading goes. i agree we really want to
upgrade.
> until i
> >>> >> can
> >>> >> >>> find a work around for the config-delete-queues deserialization
> bug
> >>> >> >>> introduced in 2.2.0 i brought up back in April we
will not be
> able
> >>> to
> >>> >> >>> easily move.
> >>> >> >>
> >>> >> >> You should be able to move to 2.6.2. if you're not able
to I
> would
> >>> >> >> like to know where it failed.
> >>> >> >>
> >>> >> >>
> >>> >> >>>
> >>> >> >>> thanks again for all the help
> >>> >> >>>
> >>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
> >>> >> clebert.suconic@gmail.com>
> >>> >> >>> wrote:
> >>> >> >>>
> >>> >> >>>> If you do not want to upgrade for any reason export
the
> journal.
> >>> >> Cleani
> >>> >> >>>> uo.  Edit the text and remove the garbage (you
will see)
> manually.
> >>> >>  Delete
> >>> >> >>>> all data and te import.
> >>> >> >>>>
> >>> >> >>>> (Make a backup to be safe of course)
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >>>> But I still recommend the upgrade.
> >>> >> >>>>
> >>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic
<
> >>> >> >>>> clebert.suconic@gmail.com>
> >>> >> >>>> wrote:
> >>> >> >>>>
> >>> >> >>>> > The address setting is the garbage I was
talking about.
> Upgrade
> >>> to
> >>> >> the
> >>> >> >>>> > latest broker and there will be a cleanup
done at the load
> >>> before it
> >>> >> >>>> > starts.
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> > I highly recommend upgrade.
> >>> >> >>>> >
> >>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram
<
> >>> >> jbertram@apache.org>
> >>> >> >>>> > wrote:
> >>> >> >>>> >
> >>> >> >>>> >> Analyzing thread dumps like this is pretty
simple.  I
> generally
> >>> >> just
> >>> >> >>>> >> scroll
> >>> >> >>>> >> through and look for long stack-traces
with lots of calls
> from
> >>> >> >>>> >> org.apache.activemq.artemis.  In your
case every single
> thread
> >>> >> dump has
> >>> >> >>>> a
> >>> >> >>>> >> thread doing something like this:
> >>> >> >>>> >>
> >>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20
nid=0x74fe
> >>> >> runnable
> >>> >> >>>> >> [0x00007f9031675000]
> >>> >> >>>> >>    java.lang.Thread.State: RUNNABLE
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
> >>> >> >>>> >>     - locked <0x0000000080a8fce8>
(a
> >>> >> >>>> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
> >>> >> >>>> >>     - locked <0x0000000080a8fce8>
(a
> >>> >> >>>> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>> >> >>>> >>     - locked <0x000000008098be80>
(a
> >>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
> >>> >> >>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>> Method)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> >> >>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
> >>> >> >>>> >>     at
> >>> >> >>>>
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>> >> >>>> >>
> >>> >> >>>> >> In every case it's the "main" thread
which isn't surprising
> as
> >>> >> that is
> >>> >> >>>> the
> >>> >> >>>> >> thread responsible for starting the broker.
 Also, you can
> >>> pretty
> >>> >> >>>> clearly
> >>> >> >>>> >> see in the trace that this is the thread
starting the
> broker,
> >>> it's
> >>> >> >>>> loading
> >>> >> >>>> >> the journals, & restoring stored
configuration (either
> address
> >>> >> settings
> >>> >> >>>> or
> >>> >> >>>> >> security settings).  I've seen high broker
start times when
> >>> there
> >>> >> are
> >>> >> >>>> lots
> >>> >> >>>> >> and lots of addresses and lots of and
lots of settings.  Do
> >>> either
> >>> >> (or
> >>> >> >>>> >> both) of these situations apply to you?
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>> >> Justin
> >>> >> >>>> >>
> >>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan
Langford <
> >>> >> danlangford@gmail.com>
> >>> >> >>>> >> wrote:
> >>> >> >>>> >>
> >>> >> >>>> >> > Dang I can’t easily upgrade past
2.1.0 because of the
> >>> >> >>>> >> config-delete-queues
> >>> >> >>>> >> > deserialization bug introduced in
2.2.0. Unless that bug
> was
> >>> >> squashed
> >>> >> >>>> in
> >>> >> >>>> >> > 2.6+. I don’t think I made a jira
for it (vacation and
> work
> >>> >> load) but
> >>> >> >>>> we
> >>> >> >>>> >> > discussed it back in April. I should
go confirm that bug
> on
> >>> 2.6
> >>> >> and
> >>> >> >>>> >> make a
> >>> >> >>>> >> > jira for that
> >>> >> >>>> >> >
> >>> >> >>>> >> > Thanks
> >>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM
Clebert Suconic <
> >>> >> >>>> >> clebert.suconic@gmail.com
> >>> >> >>>> >> > >
> >>> >> >>>> >> > wrote:
> >>> >> >>>> >> >
> >>> >> >>>> >> > > There is an issue I remember
where the journal would
> have
> >>> some
> >>> >> dirt
> >>> >> >>>> >> that
> >>> >> >>>> >> > > was fixed on 2.3/0.
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > I would ipgrade to 2.6.2.
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34
PM Dan Langford <
> >>> >> danlangford@gmail.com
> >>> >> >>>> >
> >>> >> >>>> >> > > wrote:
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > > would you be willing to
help me translate these thread
> >>> dumps?
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > i attached a Zip file
with some thread dumps in them.
> i
> >>> will
> >>> >> also
> >>> >> >>>> >> share
> >>> >> >>>> >> > > > the fasthread.io links
for each file. (i was
> struggling
> >>> >> getting
> >>> >> >>>> >> > > > fastthread to do a combo
report with the threads in
> the
> >>> >> correct
> >>> >> >>>> >> order)
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > artemis04-20180719-1525
https://goo.gl/d88azU
> >>> >> >>>> >> > > > artemis04-20180719-1530
https://goo.gl/G78qn3
> >>> >> >>>> >> > > > artemis04-20180719-1535
https://goo.gl/aMBSBw
> >>> >> >>>> >> > > > artemis04-20180719-1540
https://goo.gl/brKxxk
> >>> >> >>>> >> > > > artemis04-20180719-1545
https://goo.gl/RaXXCs
> >>> >> >>>> >> > > > artemis04-20180719-1550
https://goo.gl/r5dndK
> >>> >> >>>> >> > > > artemis04-20180719-1555
https://goo.gl/YJRLxe
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > at :35, :45, and :55 the
young+old gen space gets
> bigger
> >>> >> than at
> >>> >> >>>> the
> >>> >> >>>> >> > > other
> >>> >> >>>> >> > > > sample times. but i dont
know what to look for in
> here to
> >>> >> >>>> determine
> >>> >> >>>> >> > what
> >>> >> >>>> >> > > > the broker is actually
during during this time.
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > thanks
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > On Fri, Jul 6, 2018 at
10:42 AM Justin Bertram <
> >>> >> >>>> jbertram@apache.org
> >>> >> >>>> >> >
> >>> >> >>>> >> > > > wrote:
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > >> The first place I
would start is grabbing thread
> dumps
> >>> >> every so
> >>> >> >>>> >> often
> >>> >> >>>> >> > to
> >>> >> >>>> >> > > >> see what the broker
is actually doing during the 30+
> >>> >> minutes.
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> Justin
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> On Fri, Jul 6, 2018
at 11:34 AM, Dan Langford <
> >>> >> >>>> >> danlangford@gmail.com>
> >>> >> >>>> >> > > >> wrote:
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> > so my server
startup times and failover times are
> >>> growing
> >>> >> >>>> pretty
> >>> >> >>>> >> > big.
> >>> >> >>>> >> > > >> but i
> >>> >> >>>> >> > > >> > dont really know
where to start looking.
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > here is a snippet
of some logs to show you the time
> >>> >> stamps:
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:31,801
INFO  [org.apache.activemq.artemis.
> >>> >> >>>> >> > integration.bootstrap]
> >>> >> >>>> >> > > >> > AMQ101000: Starting
ActiveMQ Artemis Server
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:31,818
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221000:
> >>> >> >>>> >> > > >> > live Message
Broker is starting with configuration
> >>> Broker
> >>> >> >>>> >> > > Configuration
> >>> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
> >>> >> >>>> >> > journal,bindingsDirectory=./
> >>> >> >>>> >> > > >> >
> >>> >> data/bindings,largeMessagesDirectory=./data/large-messages,
> >>> >> >>>> >> > > >> > pagingDirectory=./data/paging)
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,462
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221012:
> >>> >> >>>> >> > > >> > Using AIO Journal
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,493
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221057:
> >>> >> >>>> >> > > >> > Global Max Size
is being adjusted to 1/2 of the
> JVM max
> >>> >> size
> >>> >> >>>> >> (-Xmx).
> >>> >> >>>> >> > > >> being
> >>> >> >>>> >> > > >> > defined as 1,073,741,824
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,555
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module
found: [artemis-server]. Adding
> >>> protocol
> >>> >> >>>> support
> >>> >> >>>> >> > for:
> >>> >> >>>> >> > > >> CORE
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,555
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module
found: [artemis-amqp-protocol].
> Adding
> >>> >> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: AMQP
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module
found: [artemis-hornetq-protocol].
> >>> Adding
> >>> >> >>>> >> protocol
> >>> >> >>>> >> > > >> support
> >>> >> >>>> >> > > >> > for: HORNETQ
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module
found: [artemis-mqtt-protocol].
> Adding
> >>> >> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: MQTT
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module
found: [artemis-openwire-protocol].
> >>> Adding
> >>> >> >>>> >> protocol
> >>> >> >>>> >> > > >> support
> >>> >> >>>> >> > > >> > for: OPENWIRE
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556
INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module
found: [artemis-stomp-protocol].
> Adding
> >>> >> >>>> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: STOMP
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,963
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address
configured for queue DLQ in
> >>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,963
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address
configured for queue DLQ in
> >>> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,983
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address
configured for queue
> ExpiryQueue in
> >>> >> >>>> >> > > AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,983
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address
configured for queue ExpiryQueue in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,984
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address
configured for queue example in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,985
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address
configured for queue example in
> >>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,985
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address
configured for queue
> exampleQueue
> >>> in
> >>> >> >>>> >> > > AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,986
WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address
configured for queue exampleQueue in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > and it continues.
i have 138 queues. i wonder if i
> need
> >>> >> to be
> >>> >> >>>> >> > looking
> >>> >> >>>> >> > > at
> >>> >> >>>> >> > > >> > PAGE configuration
or some cache sizes. do i need
> to be
> >>> >> looking
> >>> >> >>>> >> at
> >>> >> >>>> >> > the
> >>> >> >>>> >> > > >> > number of messages
persisted on these queues? where
> >>> would
> >>> >> you
> >>> >> >>>> >> look
> >>> >> >>>> >> > to
> >>> >> >>>> >> > > >> > determine why
the startup times are so long?
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > also, i know
i need to upgrade but i cannot upgrade
> >>> off of
> >>> >> >>>> 2.1.0
> >>> >> >>>> >> due
> >>> >> >>>> >> > > to
> >>> >> >>>> >> > > >> > some deseralization
changes introduced in 2.2.0. i
> >>> think
> >>> >> i have
> >>> >> >>>> >> > > another
> >>> >> >>>> >> > > >> > thread on here
(that i need to update) regarding
> that
> >>> >> upgrade
> >>> >> >>>> >> issue.
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > thanks for any
ideas or insight you have for me
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > > --
> >>> >> >>>> >> > > Clebert Suconic
> >>> >> >>>> >> > >
> >>> >> >>>> >> >
> >>> >> >>>> >>
> >>> >> >>>> > --
> >>> >> >>>> > Clebert Suconic
> >>> >> >>>> >
> >>> >> >>>> --
> >>> >> >>>> Clebert Suconic
> >>> >> >>>>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> --
> >>> >> >> Clebert Suconic
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > Clebert Suconic
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Clebert Suconic
> >>> >>
> >>>
> >>>
> >>>
> >>> --
> >>> Clebert Suconic
> >>>
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message