From users-return-50395-archive-asf-public=cust-asf.ponee.io@activemq.apache.org Thu Aug 23 01:28:58 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4C4D1180662 for ; Thu, 23 Aug 2018 01:28:57 +0200 (CEST) Received: (qmail 76825 invoked by uid 500); 22 Aug 2018 23:28:51 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 76812 invoked by uid 99); 22 Aug 2018 23:28:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Aug 2018 23:28:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6673BC0115 for ; Wed, 22 Aug 2018 23:28:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.89 X-Spam-Level: * X-Spam-Status: No, score=1.89 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id KXTp50fodV7C for ; Wed, 22 Aug 2018 23:28:42 +0000 (UTC) Received: from mail-pl0-f42.google.com (mail-pl0-f42.google.com [209.85.160.42]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 4589C5F2AD for ; Wed, 22 Aug 2018 23:28:42 +0000 (UTC) Received: by mail-pl0-f42.google.com with SMTP id g23-v6so1519435plq.9 for ; Wed, 22 Aug 2018 16:28:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Md+sYyRcTYBmhtohlxOKMPNAPffOA1SZkvFiXYd8Vic=; b=PHix9Rc6hrQ1joQ3LjuW0u/CDVvLafPdwW5YuZOSGUKMifO+3+/V6YKxAiQ9IDYpz2 uRYCWCj4CnlxM+iBeGKqSS+jfcmx7g8LkNHhj8IS65H+8EqS8DJh0QSzZibaycDwFkG8 ISgzQ5erWZJopFNOmB1LgzP74ca4ncjnG/i91BY5K8utnhFDqjXvqy6xiJzTKGSjOAOm jeayds5aiwHa/eqg+jBDCfRalzxrQBA09E8gMPI0ltoflTlbrIN5Y6waf3vFwch97WXO /9g6wSzATSLoAM2u6Ar/L2tAT6rlVu6naXChVfq59yL00zqhYLVEvHXscmMi59QNyTO3 5PbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Md+sYyRcTYBmhtohlxOKMPNAPffOA1SZkvFiXYd8Vic=; b=ZjrEp7AOA4uN6G0VDl7+XKYUfMsBpZ6fDitcKuCPxUcvDjH9xMQEHdOsAq1omJZVKN BHBT644vupyhnYNmNNMAVcqJHo8nDIz1Wfxjj7KZKtTZQPLmYW7/fAnsMHJqqzhTogJf yjgC3u5IBhyvR01xjNP99H3cEtYy+CiJphLKBeaK5lFLoRY+xW2v2h9qzt5JbwS8/WVV R+q9bNixWM4x3QqikEkfV6vjWxrfyMkjXjEY8DkBGgQO3mlt6G2v8xGDbUInJSp/bEKr ebGOtPpGPy8l1idIG2NOtPlGwLDYyCbaC+lRGWclhOKzFXiXt6D2AlLmcUZXO62l085+ Uxng== X-Gm-Message-State: AOUpUlELzweg8KbY1IhW2RMKdGNz40EF9kBEn6Lfmo0pDtWM07ycFMJn Keo3gnfX8eD/MKniL5y9mOaHCtYkeBFIMm9hzNKO5A== X-Google-Smtp-Source: AA+uWPxQLJAcn/mHqElMzaUYZ244srFLo4/eNHaLQtyi7JA8HoP006OS+s8AtQcSLIykKlMARqDW2GsA5w5zxgxFNbo= X-Received: by 2002:a17:902:7587:: with SMTP id j7-v6mr56061694pll.256.1534980520742; Wed, 22 Aug 2018 16:28:40 -0700 (PDT) MIME-Version: 1.0 References: <0PCF00IRGWADCY10@mr11p00im-asmtp001.me.com> In-Reply-To: From: Dan Langford Date: Wed, 22 Aug 2018 17:28:28 -0600 Message-ID: Subject: Re: [artemis 2.1.0] taking 30+ minutes to boot & failover To: users@activemq.apache.org Content-Type: multipart/alternative; boundary="000000000000ad13ba05740e80d0" --000000000000ad13ba05740e80d0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable i validated that the fix did in fact fix our upgrade path. i made a new 2.1.0 broker. i added some address settings via JMX. i attempted an upgrade to 2.6.2 and watched it fail. i pulled the code that you linked above. i packaged a release and then i upgraded to the released code and the server started just fine. thank you so much!! what is the timeline on a 2.6.3 or 2.7.0 release? On Fri, Aug 3, 2018 at 4:22 PM Clebert Suconic wrote: > I'm considering doing a 2.6.3 very soon.. it will include the fix.. > you should be able to move from 2.1.0 . let me know if you have any > other issues. > > On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic > wrote: > > https://github.com/apache/activemq-artemis/pull/2214 > > > > > > I will make it into 2.6.x as soon as the PR tests are finished. > > > > > > If you could try a snapshot of master or 2.6.x (after I merge it of > > course). Let me know if you need help buildling it. > > > > On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford > wrote: > >> Michael for the win! > >> @Clebert im not opposed to coding up a test case it is just that i am > >> swamped at the moment so im glad to hear that Michael already has a > bead on > >> this > >> > >> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis da= ta > >> imp* on a clean 2.6.2. the new system didnt have Address Settings and > >> Security Settings that had previously been added via the jolokia api. = i > >> should look into the export options more to see if i just failed to > provide > >> a command line argument. fortunately in my scenario it was pretty > trivial > >> to recreate those settings quickly. so i did that. the import created > all > >> the addresses, routes, queues, and messages that i needed. it went > pretty > >> well. so my cluster is now on 2.6.2 > >> > >> i plan to rework our usage of AddressSetting to not create a new entry > for > >> every Address we create but instead only create them for the few > exceptions > >> to our default configuration > >> > >> then i can start working on running the broker in CF with the cluster = in > >> more of a Master-Master mode. > >> thanks again for all your help. feel free to respond if there is any > more > >> data you need from me. > >> > >> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds= ) > >> while handshaking has occurred. " but thats another issue i will figur= e > >> out. * > >> > >> my 30 minute startup times are now down to 30 seconds max. more like > 10-15 > >> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP. > >> > >> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce > >> wrote: > >> > >>> Clebert im able to recreate this and also worked out in code why this > >>> happens. Ill try catch you on irc tomorrow or friday. > >>> > >>> Sent from my Samsung Galaxy smartphone. > >>> -------- Original message --------From: Clebert Suconic < > >>> clebert.suconic@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To: > >>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ > minutes > >>> to boot & failover > >>> Yes.. it should include it. > >>> > >>> > >>> Do you have an example of the API you used with 2.1? > >>> > >>> > >>> I can't write a compatibility test this week.. but if you provide me > >>> the example I will do it early next week. > >>> > >>> > >>> In a perfect world, if you did it .. it would be great :) > >>> > >>> > >>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford > >>> wrote: > >>> > i tried 2.6.2 this morning to see if that was an improvement from > 2.5.0 i > >>> > tried a few months ago. unfortunately there was not. it may have > failed > >>> > much faster than 2.5.0 i dont recall the timing specifics but the > error > >>> is > >>> > the same > >>> > > >>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] > AMQ224000: > >>> > Failure in initialisation: java.lang.NegativeArraySizeException > >>> > at > >>> > > >>> > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(Simple= String.java:182) > >>> > [artemis-commons-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(Simple= String.java:171) > >>> > [artemis-commons-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleStrin= g(SimpleString.java:158) > >>> > [artemis-commons-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNu= llableSimpleString(ChannelBufferWrapper.java:69) > >>> > [artemis-commons-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(Add= ressSettings.java:736) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetti= ng.decode(PersistedAddressSetting.java:95) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournal= StorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournal= StorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJourn= als(ActiveMQServerImpl.java:2601) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialis= ePart2(ActiveMQServerImpl.java:2376) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.= run(SharedNothingLiveActivation.java:116) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalS= tart(ActiveMQServerImpl.java:535) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(Act= iveMQServerImpl.java:474) > >>> > [artemis-server-2.6.2.jar:2.6.2] > >>> > at > >>> > > >>> > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:= 111) > >>> > [artemis-cli-2.6.2.jar:2.6.2] > >>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82= ) > >>> > [artemis-cli-2.6.2.jar:2.6.2] > >>> > at > >>> > > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149) > >>> > [artemis-cli-2.6.2.jar:2.6.2] > >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97) > >>> > [artemis-cli-2.6.2.jar:2.6.2] > >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124= ) > >>> > [artemis-cli-2.6.2.jar:2.6.2] > >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> > [rt.jar:1.8.0_171] > >>> > at > >>> > > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :62) > >>> > [rt.jar:1.8.0_171] > >>> > at > >>> > > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:43) > >>> > [rt.jar:1.8.0_171] > >>> > at java.lang.reflect.Method.invoke(Method.java:498) > [rt.jar:1.8.0_171] > >>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:12= 9) > >>> > [artemis-boot.jar:2.6.2] > >>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > >>> > [artemis-boot.jar:2.6.2] > >>> > > >>> > > >>> > right there in the AddressSettings.decode > >>> > > >>> > > >>> > I am going to start looking into a journal export and then a journa= l > >>> import > >>> > and see what happens there. > >>> > are programmatically created addresses / queues stored in the data > export > >>> > along with the address and security settings? > >>> > > >>> > > >>> > > >>> > > >>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic < > >>> clebert.suconic@gmail.com> > >>> > wrote: > >>> > > >>> >> @dan take a look at the compatibility tests in master.. We're help > to > >>> >> help if you don't understand anything.. it's using ClassLoaders an= d > >>> >> Groovy so you can produce stuff with a combination of clients and > >>> >> servers on 2.1.0 and consume on the current version. > >>> >> > >>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic > >>> >> wrote: > >>> >> > @Dan: what about this: > >>> >> > > >>> >> > you provide us a code path that could generate the > incompatibility in > >>> >> 2.1.0... > >>> >> > > >>> >> > an ultimate deal would be if you produced a compatibility test > with > >>> >> > 2.1.0.... you add some code that will generate the journal with > 2.1.0, > >>> >> > and then consume the messages on master (or 2.6.x)... > >>> >> > > >>> >> > > >>> >> > if you generate the compatibility test showing the issue, we can > try > >>> >> fixing it. > >>> >> > > >>> >> > > >>> >> > What about this idea? that would help you migrate into 2.6.x or > >>> master. > >>> >> > > >>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic > >>> >> > wrote: > >>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford < > danlangford@gmail.com > >>> > > >>> >> wrote: > >>> >> >>> Thank you that was very helpful. we actually do have an addres= s > >>> >> settings > >>> >> >>> entry for each queue. there could be a better pattern for us. > but > >>> >> currently > >>> >> >>> our automated system for creating queues creates an address > setting > >>> at > >>> >> the > >>> >> >>> same time. i will look into improved patterns. > >>> >> >>> > >>> >> >>> as far as upgrading goes. i agree we really want to upgrade. > until i > >>> >> can > >>> >> >>> find a work around for the config-delete-queues deserializatio= n > bug > >>> >> >>> introduced in 2.2.0 i brought up back in April we will not be > able > >>> to > >>> >> >>> easily move. > >>> >> >> > >>> >> >> You should be able to move to 2.6.2. if you're not able to I > would > >>> >> >> like to know where it failed. > >>> >> >> > >>> >> >> > >>> >> >>> > >>> >> >>> thanks again for all the help > >>> >> >>> > >>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic < > >>> >> clebert.suconic@gmail.com> > >>> >> >>> wrote: > >>> >> >>> > >>> >> >>>> If you do not want to upgrade for any reason export the > journal. > >>> >> Cleani > >>> >> >>>> uo. Edit the text and remove the garbage (you will see) > manually. > >>> >> Delete > >>> >> >>>> all data and te import. > >>> >> >>>> > >>> >> >>>> (Make a backup to be safe of course) > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> But I still recommend the upgrade. > >>> >> >>>> > >>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic < > >>> >> >>>> clebert.suconic@gmail.com> > >>> >> >>>> wrote: > >>> >> >>>> > >>> >> >>>> > The address setting is the garbage I was talking about. > Upgrade > >>> to > >>> >> the > >>> >> >>>> > latest broker and there will be a cleanup done at the load > >>> before it > >>> >> >>>> > starts. > >>> >> >>>> > > >>> >> >>>> > > >>> >> >>>> > I highly recommend upgrade. > >>> >> >>>> > > >>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram < > >>> >> jbertram@apache.org> > >>> >> >>>> > wrote: > >>> >> >>>> > > >>> >> >>>> >> Analyzing thread dumps like this is pretty simple. I > generally > >>> >> just > >>> >> >>>> >> scroll > >>> >> >>>> >> through and look for long stack-traces with lots of calls > from > >>> >> >>>> >> org.apache.activemq.artemis. In your case every single > thread > >>> >> dump has > >>> >> >>>> a > >>> >> >>>> >> thread doing something like this: > >>> >> >>>> >> > >>> >> >>>> >> "main" #1 prio=3D5 os_prio=3D0 tid=3D0x00007f902800eb20 ni= d=3D0x74fe > >>> >> runnable > >>> >> >>>> >> [0x00007f9031675000] > >>> >> >>>> >> java.lang.Thread.State: RUNNABLE > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectReposito= ry.getPossibleMatches(HierarchicalObjectRepository.java:373) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectReposito= ry.getMatch(HierarchicalObjectRepository.java:192) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySet= tings(PagingManagerImpl.java:113) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(P= agingManagerImpl.java:108) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectReposito= ry.onChange(HierarchicalObjectRepository.java:348) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectReposito= ry.addMatch(HierarchicalObjectRepository.java:168) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectReposito= ry.addMatch(HierarchicalObjectRepository.java:147) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectReposito= ry.addMatch(HierarchicalObjectRepository.java:120) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverSt= oredConfigs(ActiveMQServerImpl.java:2424) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJourn= als(ActiveMQServerImpl.java:2374) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialis= ePart2(ActiveMQServerImpl.java:2219) > >>> >> >>>> >> - locked <0x0000000080a8fce8> (a > >>> >> >>>> >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.= run(SharedNothingLiveActivation.java:109) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalS= tart(ActiveMQServerImpl.java:518) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(Act= iveMQServerImpl.java:466) > >>> >> >>>> >> - locked <0x0000000080a8fce8> (a > >>> >> >>>> >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:= 111) > >>> >> >>>> >> - locked <0x000000008098be80> (a > >>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker) > >>> >> >>>> >> at > >>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) > >>> >> >>>> >> at > >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) > >>> >> >>>> >> at > >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) > >>> >> >>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>> Method) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :62) > >>> >> >>>> >> at > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> > >>> >> > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:43) > >>> >> >>>> >> at java.lang.reflect.Method.invoke(Method.java:498) > >>> >> >>>> >> at > >>> >> >>>> > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > >>> >> >>>> >> at > >>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > >>> >> >>>> >> > >>> >> >>>> >> In every case it's the "main" thread which isn't surprisin= g > as > >>> >> that is > >>> >> >>>> the > >>> >> >>>> >> thread responsible for starting the broker. Also, you can > >>> pretty > >>> >> >>>> clearly > >>> >> >>>> >> see in the trace that this is the thread starting the > broker, > >>> it's > >>> >> >>>> loading > >>> >> >>>> >> the journals, & restoring stored configuration (either > address > >>> >> settings > >>> >> >>>> or > >>> >> >>>> >> security settings). I've seen high broker start times whe= n > >>> there > >>> >> are > >>> >> >>>> lots > >>> >> >>>> >> and lots of addresses and lots of and lots of settings. D= o > >>> either > >>> >> (or > >>> >> >>>> >> both) of these situations apply to you? > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> >> Justin > >>> >> >>>> >> > >>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford < > >>> >> danlangford@gmail.com> > >>> >> >>>> >> wrote: > >>> >> >>>> >> > >>> >> >>>> >> > Dang I can=E2=80=99t easily upgrade past 2.1.0 because o= f the > >>> >> >>>> >> config-delete-queues > >>> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug > was > >>> >> squashed > >>> >> >>>> in > >>> >> >>>> >> > 2.6+. I don=E2=80=99t think I made a jira for it (vacati= on and > work > >>> >> load) but > >>> >> >>>> we > >>> >> >>>> >> > discussed it back in April. I should go confirm that bug > on > >>> 2.6 > >>> >> and > >>> >> >>>> >> make a > >>> >> >>>> >> > jira for that > >>> >> >>>> >> > > >>> >> >>>> >> > Thanks > >>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic < > >>> >> >>>> >> clebert.suconic@gmail.com > >>> >> >>>> >> > > > >>> >> >>>> >> > wrote: > >>> >> >>>> >> > > >>> >> >>>> >> > > There is an issue I remember where the journal would > have > >>> some > >>> >> dirt > >>> >> >>>> >> that > >>> >> >>>> >> > > was fixed on 2.3/0. > >>> >> >>>> >> > > > >>> >> >>>> >> > > I would ipgrade to 2.6.2. > >>> >> >>>> >> > > > >>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford < > >>> >> danlangford@gmail.com > >>> >> >>>> > > >>> >> >>>> >> > > wrote: > >>> >> >>>> >> > > > >>> >> >>>> >> > > > would you be willing to help me translate these thre= ad > >>> dumps? > >>> >> >>>> >> > > > > >>> >> >>>> >> > > > i attached a Zip file with some thread dumps in them= . > i > >>> will > >>> >> also > >>> >> >>>> >> share > >>> >> >>>> >> > > > the fasthread.io links for each file. (i was > struggling > >>> >> getting > >>> >> >>>> >> > > > fastthread to do a combo report with the threads in > the > >>> >> correct > >>> >> >>>> >> order) > >>> >> >>>> >> > > > > >>> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU > >>> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3 > >>> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw > >>> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk > >>> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs > >>> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK > >>> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe > >>> >> >>>> >> > > > > >>> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets > bigger > >>> >> than at > >>> >> >>>> the > >>> >> >>>> >> > > other > >>> >> >>>> >> > > > sample times. but i dont know what to look for in > here to > >>> >> >>>> determine > >>> >> >>>> >> > what > >>> >> >>>> >> > > > the broker is actually during during this time. > >>> >> >>>> >> > > > > >>> >> >>>> >> > > > thanks > >>> >> >>>> >> > > > > >>> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram < > >>> >> >>>> jbertram@apache.org > >>> >> >>>> >> > > >>> >> >>>> >> > > > wrote: > >>> >> >>>> >> > > > > >>> >> >>>> >> > > >> The first place I would start is grabbing thread > dumps > >>> >> every so > >>> >> >>>> >> often > >>> >> >>>> >> > to > >>> >> >>>> >> > > >> see what the broker is actually doing during the 30= + > >>> >> minutes. > >>> >> >>>> >> > > >> > >>> >> >>>> >> > > >> > >>> >> >>>> >> > > >> Justin > >>> >> >>>> >> > > >> > >>> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford < > >>> >> >>>> >> danlangford@gmail.com> > >>> >> >>>> >> > > >> wrote: > >>> >> >>>> >> > > >> > >>> >> >>>> >> > > >> > so my server startup times and failover times are > >>> growing > >>> >> >>>> pretty > >>> >> >>>> >> > big. > >>> >> >>>> >> > > >> but i > >>> >> >>>> >> > > >> > dont really know where to start looking. > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > here is a snippet of some logs to show you the ti= me > >>> >> stamps: > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis. > >>> >> >>>> >> > integration.bootstrap] > >>> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:31,818 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221000: > >>> >> >>>> >> > > >> > live Message Broker is starting with configuratio= n > >>> Broker > >>> >> >>>> >> > > Configuration > >>> >> >>>> >> > > >> > (clustered=3Dtrue,journalDirectory=3D./data/ > >>> >> >>>> >> > journal,bindingsDirectory=3D./ > >>> >> >>>> >> > > >> > > >>> >> data/bindings,largeMessagesDirectory=3D./data/large-messages, > >>> >> >>>> >> > > >> > pagingDirectory=3D./data/paging) > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,462 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221012: > >>> >> >>>> >> > > >> > Using AIO Journal > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,493 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221057: > >>> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the > JVM max > >>> >> size > >>> >> >>>> >> (-Xmx). > >>> >> >>>> >> > > >> being > >>> >> >>>> >> > > >> > defined as 1,073,741,824 > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,555 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221043: > >>> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding > >>> protocol > >>> >> >>>> support > >>> >> >>>> >> > for: > >>> >> >>>> >> > > >> CORE > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,555 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221043: > >>> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. > Adding > >>> >> protocol > >>> >> >>>> >> > > support > >>> >> >>>> >> > > >> > for: AMQP > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,556 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221043: > >>> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]= . > >>> Adding > >>> >> >>>> >> protocol > >>> >> >>>> >> > > >> support > >>> >> >>>> >> > > >> > for: HORNETQ > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,556 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221043: > >>> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. > Adding > >>> >> protocol > >>> >> >>>> >> > > support > >>> >> >>>> >> > > >> > for: MQTT > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,556 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221043: > >>> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol= ]. > >>> Adding > >>> >> >>>> >> protocol > >>> >> >>>> >> > > >> support > >>> >> >>>> >> > > >> > for: OPENWIRE > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:11:34,556 INFO > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ221043: > >>> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. > Adding > >>> >> >>>> protocol > >>> >> >>>> >> > > support > >>> >> >>>> >> > > >> > for: STOMP > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,963 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222165: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in > >>> >> AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,963 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222166: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in > >>> AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,983 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222165: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Dead Letter Address configured for queue > ExpiryQueue in > >>> >> >>>> >> > > AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,983 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222166: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue i= n > >>> >> >>>> >> AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,984 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222165: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Dead Letter Address configured for queue example = in > >>> >> >>>> >> AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,985 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222166: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Expiry Address configured for queue example in > >>> >> AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,985 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222165: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Dead Letter Address configured for queue > exampleQueue > >>> in > >>> >> >>>> >> > > AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > 08:41:38,986 WARN > >>> >> [org.apache.activemq.artemis.core.server] > >>> >> >>>> >> > > AMQ222166: > >>> >> >>>> >> > > >> No > >>> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue = in > >>> >> >>>> >> AddressSettings > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if = i > need > >>> >> to be > >>> >> >>>> >> > looking > >>> >> >>>> >> > > at > >>> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need > to be > >>> >> looking > >>> >> >>>> >> at > >>> >> >>>> >> > the > >>> >> >>>> >> > > >> > number of messages persisted on these queues? whe= re > >>> would > >>> >> you > >>> >> >>>> >> look > >>> >> >>>> >> > to > >>> >> >>>> >> > > >> > determine why the startup times are so long? > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgra= de > >>> off of > >>> >> >>>> 2.1.0 > >>> >> >>>> >> due > >>> >> >>>> >> > > to > >>> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. = i > >>> think > >>> >> i have > >>> >> >>>> >> > > another > >>> >> >>>> >> > > >> > thread on here (that i need to update) regarding > that > >>> >> upgrade > >>> >> >>>> >> issue. > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > thanks for any ideas or insight you have for me > >>> >> >>>> >> > > >> > > >>> >> >>>> >> > > >> > >>> >> >>>> >> > > > -- > >>> >> >>>> >> > > Clebert Suconic > >>> >> >>>> >> > > > >>> >> >>>> >> > > >>> >> >>>> >> > >>> >> >>>> > -- > >>> >> >>>> > Clebert Suconic > >>> >> >>>> > > >>> >> >>>> -- > >>> >> >>>> Clebert Suconic > >>> >> >>>> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> -- > >>> >> >> Clebert Suconic > >>> >> > > >>> >> > > >>> >> > > >>> >> > -- > >>> >> > Clebert Suconic > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Clebert Suconic > >>> >> > >>> > >>> > >>> > >>> -- > >>> Clebert Suconic > >>> > > > > > > > > -- > > Clebert Suconic > > > > -- > Clebert Suconic > --000000000000ad13ba05740e80d0--