Mailing-List: contact issues-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Date: Mon, 8 Aug 2016 12:43:20 +0000 (UTC)
From: "Czeslaw (JIRA)" <jira@apache.org>
To: issues@activemq.apache.org
Message-ID: <JIRA.12916971.1448894532000.248688.1470660200508@Atlassian.JIRA>
In-Reply-To: <JIRA.12916971.1448894532000@Atlassian.JIRA>
References: <JIRA.12916971.1448894532000@Atlassian.JIRA> <JIRA.12916971.1448894532643@arcas>
Subject: [jira] [Commented] (AMQ-6067) OutOfMemoryError when expiring big
 amount of topic messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 08 Aug 2016 12:43:22 -0000


    [ https://issues.apache.org/jira/browse/AMQ-6067?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D154117=
46#comment-15411746 ]=20

Czeslaw commented on AMQ-6067:
------------------------------

Question is how we can mitigate this memory issue for system where is runni=
ng amq 5.13.2. I  can  try to increase =E2=80=93Xmx, make smaller expireMes=
sagesPeriods so we will expire messages more frequently. Do you see other o=
ptions ?

> OutOfMemoryError when expiring big amount of topic messages
> -----------------------------------------------------------
>
>                 Key: AMQ-6067
>                 URL: https://issues.apache.org/jira/browse/AMQ-6067
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: JDBC
>    Affects Versions: 5.10.0
>            Reporter: Petr Havr=C3=A1nek
>              Labels: durable, durable_subscription, expiration, jdbc, tim=
eToLive,
>
> There is a problem in
> {noformat}
> org.apache.activemq.broker.region.Topic.expireMessagesTask
> {noformat}
> When there are big amount of topic messages that are going to be expired,=
 this {{expireMessagesTask}} loads all of the messages to memory. This caus=
es
> {noformat}
> 2015-11-24 11:05:46.359 WARN  [ActiveMQ Broker[JmsEngineActivemqBroker] S=
cheduler] [Topic] Failed to browse Topic: test-topic
> java.lang.OutOfMemoryError: Java heap space
> =09at oracle.sql.BLOB.getBytes(BLOB.java:204)
> =09at oracle.jdbc.driver.T4CBlobAccessor.getBytes(T4CBlobAccessor.java:46=
4)
> =09at oracle.jdbc.driver.OracleResultSetImpl.getBytes(OracleResultSetImpl=
.java:676)
> =09at org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResu=
ltSet.java:203)
> =09at org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getBinary=
Data(DefaultJDBCAdapter.java:80)
> =09at org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doRecover=
(DefaultJDBCAdapter.java:418)
> =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp=
l.java:39)
> =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc=
essorImpl.java:25)
> =09at java.lang.reflect.Method.invoke(Method.java:597)
> =09at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflec=
tion(AopUtils.java:309)
> =09at org.springframework.aop.framework.ReflectiveMethodInvocation.invoke=
Joinpoint(ReflectiveMethodInvocation.java:183)
> =09at org.springframework.aop.framework.ReflectiveMethodInvocation.procee=
d(ReflectiveMethodInvocation.java:150)
> =09at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke=
(AbstractTraceInterceptor.java:113)
> =09at org.springframework.aop.framework.ReflectiveMethodInvocation.procee=
d(ReflectiveMethodInvocation.java:172)
> =09at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDyna=
micAopProxy.java:202)
> =09at $Proxy14.doRecover(Unknown Source)
> =09at org.apache.activemq.store.jdbc.JDBCMessageStore.recover(JDBCMessage=
Store.java:236)
> =09at org.apache.activemq.store.ProxyTopicMessageStore.recover(ProxyTopic=
MessageStore.java:62)
> =09at org.apache.activemq.broker.region.Topic.doBrowse(Topic.java:594)
> =09at org.apache.activemq.broker.region.Topic.access$100(Topic.java:65)
> =09at org.apache.activemq.broker.region.Topic$6.run(Topic.java:733)
> =09at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTas=
k.java:33)
> =09at java.util.TimerThread.mainLoop(Timer.java:512)
> =09at java.util.TimerThread.run(Timer.java:462)
> {noformat}
> The problem happens when using JDBC persistency with ActiveMQ 5.10.0. Aft=
er a short look to source code, the same problem could be also with 5.12.1.
> Test case:
> - run ActiveMQ broker with JDBC persistency
> - create subscription to a topic, but do not receive the messages
> - send enough number of messages with short TimeToLive
> - when expireMessagesTask is scheduled, it tries to load all of the messa=
ges and causes the OutOfMemoryError
> It would be fine if
> {noformat}
> org.apache.activemq.store.jdbc.JDBCMessageStore.recover(MessageRecoveryLi=
stener)
> {noformat}
> will be updated like this:
> {code:java}
> public void recover(final MessageRecoveryListener listener) throws Except=
ion {
>   // Get all the Message ids out of the database.
>   TransactionContext c =3D persistenceAdapter.getTransactionContext();
>   try {
>     c =3D persistenceAdapter.getTransactionContext();
>     adapter.doRecover(c, destination, new JDBCMessageRecoveryListener() {
>       public boolean recoverMessage(long sequenceId, byte[] data) throws =
Exception {
>         if (listener.hasSpace()) {
>           Message msg =3D (Message)wireFormat.unmarshal(new ByteSequence(=
data));
>           msg.getMessageId().setBrokerSequenceId(sequenceId);
>           return listener.recoverMessage(msg);
>         } else {
>           logger.debug("Recovery limit of the messages has exceeded.");
>           return false;
>         }                   =20
>       }
>       public boolean recoverMessageReference(String reference) throws Exc=
eption {
>         if (listener.hasSpace()) {
>           return listener.recoverMessageReference(new MessageId(reference=
));
>         } else {
>           logger.debug("Recovery limit of the message references has exce=
eded.");
>           return false;
>         }
>       }
>     });
>   } catch (SQLException e) {
>     JDBCPersistenceAdapter.log("JDBC Failure: ", e);
>     throw IOExceptionSupport.create("Failed to recover container. Reason:=
 " + e, e);
>   } finally {
>     c.close();
>   }
> }
> {code}
> But I am not sure if this limitation is the best way, because there will =
be some messages that should be expired, but need to wait. So better soluti=
on might be to do this job in more separated transactions.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)