Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
MIME-Version: 1.0
Sender: tbain98@gmail.com
In-Reply-To: <1455114112085-4707387.post@n4.nabble.com>
References: <1455114112085-4707387.post@n4.nabble.com>
Date: Fri, 12 Feb 2016 08:33:42 -0700
Message-ID: 
 <CAPVVMa82XOT_kFA+jWZSAF7J06PdXRp9MBeY1qgo_5aYKjbrag@mail.gmail.com>
Subject: Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
From: Tim Bain <tbain@alumni.duke.edu>
To: ActiveMQ Users <users@activemq.apache.org>
Content-Type: multipart/alternative; boundary=089e011765495b5806052b94645d

--089e011765495b5806052b94645d
Content-Type: text/plain; charset=UTF-8

First, for a question about GC problems, you should probably tell us what
GC strategy you're using, and why you chose it.

Second, you made the claim that "to increase the memory to 2Gb ... only
delaying the issue from happening," which implies that you believe you're
running out of memory because of a memory leak rather than because you
sized your JVM too small.  Do you have any evidence of such a leak or that
you've properly sized your JVM and run out of memory, or are you just
making an assumption?  I've spent a couple months performance testing the
5.8.0 broker's memory usage with non-persistent messages that are consumed
immediately, and I can tell you for certain that there's no memory leak in
the code paths executed by that specific scenario (in that version).  It's
always possible that there's a leak in some other code that my test wasn't
executing, but even that is somewhat unlikely given that no one but you has
complained about it.  So although it's possible that there's a leak, I
think it's more likely that either you have more live objects than will fit
into the amount of heap you've given to your JVM (in which case increasing
the heap is the solution, and a real solution), or you're using the CMS GC
strategy and you're hitting the Old Gen fragmentation problem.

Tim
On Feb 10, 2016 10:52 AM, "ealiass" <ali.assi@ericsson.com> wrote:

> Hi All,
>
> I am having java.lang.OutOfMemoryError: GC overhead limit exceeded Error
> after a while from starting the ActiveMQ on my system (2-3 weeks) start
> seeing this issue on production.
>  i had to restart the my ActiveMQ every while on production so i might lose
> some messages coming in the time while the queue is restarting.
>
> we are using the version:5.11.1
> we have 1Gb of memory for the queue on production
> there was similar issue on the forum and the answer was to increase the
> memory to 2Gb, but this is not a real solution since we only delaying the
> issue from happening
>
> find below the logs i got
>
>
> org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire,connectionViewType=remoteAddress,connectionName=tcp_//172.16.19.0_40176\n","stream":"stdout","time":"2016-01-18T10:40:55.455379802Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52528 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:40:58.87142185Z"}
> {"log":"ERROR | Could not accept connection : java.lang.Exception:
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stdout","time":"2016-01-18T10:41:00.229514014Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40222 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.994358557Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40195 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.99828402Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40193 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:06.424638043Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40374 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:09.107570289Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34082 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:11.89354178Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34072 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:13.548763654Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52575 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.788697766Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52628 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.795035469Z"}
> {"log":"ERROR | Could not accept connection : java.lang.Exception:
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stdout","time":"2016-01-18T10:41:17.821256534Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40341 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:17.824982456Z"}
> {"log":"Exception in thread \"ActiveMQ InactivityMonitor Worker\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:22.453110848Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34040 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:22.459674904Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40300 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:26.945482864Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52777 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:30.77306023Z"}
> {"log":"Exception in thread \"ActiveMQ BrokerService[localhost]
> Task-72110\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:33.768910175Z"}
> {"log":"Exception in thread \"ActiveMQ Journal Checkpoint Worker\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:34.812406659Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40283 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:37.718970346Z"}
> {"log":" INFO | KahaDB: Recovering checkpoint thread after
> death\n","stream":"stdout","time":"2016-01-18T10:41:39.163309356Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40528 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:42.018391792Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34235 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:42.019448416Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40485 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:45.595817225Z"}
> {"log":"Exception in thread \"ActiveMQ Journal Checkpoint Worker\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:47.076826925Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34264 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:50.04356476Z"}
> {"log":"Exception in thread \"ActiveMQ Transport Server:
> stomp://
> 0.0.0.0:61613?maximumConnections=1000\u0026wireFormat.maxFrameSize=104857600\
> "
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:50.047204344Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52760 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:50.049358006Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40487 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:51.04876469Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40453 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:52.941294862Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34209 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:52.943995696Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34253 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:55.257523013Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34206 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:55.25780067Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52713 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:57.147528061Z"}
> {"log":"Exception in thread \"ActiveMQ Transport:
> tcp:///172.16.90.0:34152@61616\" java.lang.OutOfMemoryError: GC overhead
> limit exceeded\n","stream":"stderr","time":"2016-01-18T10:41:57.14816742Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52717 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:57.148887172Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40437 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.59718311Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34176 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.597488872Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52671 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.598515011Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52693 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:00.585086858Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40433 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:00.586107233Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52633 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:02.988706309Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40376 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:02.992193223Z"}
> {"log":" INFO | KahaDB: Recovering checkpoint thread after
> death\n","stream":"stdout","time":"2016-01-18T10:42:04.079519366Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52821 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:04.08123258Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52859 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:04.084563477Z"}
> {"log":" WARN | Failed to register MBean
>
> if anyone one have had the same issue or have any hint to guide me please
> help would appreciate it :)
>
> Thanks,
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/java-lang-OutOfMemoryError-GC-overhead-limit-exceeded-tp4707387.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--089e011765495b5806052b94645d--