Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Date: Mon, 30 Jan 2017 06:58:13 -0800 (PST)
From: francesco81 <Francesco.PADOVANI@bticino.it>
To: users@activemq.apache.org
Message-ID: <1485788293667-4721395.post@n4.nabble.com>
In-Reply-To: <CAKF+bsproNetCcuv8gP0PrZLCi57dkPxxPAkT+K5GHf-9J8Mhw@mail.gmail.com>
References: <DB6PR0602MB2823ED3C1A022BD4AD8610D3FB720@DB6PR0602MB2823.eurprd06.prod.outlook.com> <CAKF+bsproNetCcuv8gP0PrZLCi57dkPxxPAkT+K5GHf-9J8Mhw@mail.gmail.com>
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
archived-at: Mon, 30 Jan 2017 15:00:52 -0000

Hello,
just an update about this topic.

By disabling persistence, the broker starts again to work fast and reliable.
But just for about 2 more weeks: then the memory come bake to be full and
the broker starts to page constantly, and cpu usage goes to 100%.

I tried to run the command "artemis data print" as you suggested, but I have
to admit my ignorance to read the results.
I did redirect the command result to a file and it produced about 758 MB of
data.
I noticed no strange things among the initial lines but this:

/*17:39:47,214 FINE  [io.netty.util.internal.PlatformDependent] Javassist:
unavailable
17:39:47,215 FINE  [io.netty.util.internal.PlatformDependent] You don't have
Javassist in your class path or you don't have enough permission to load
dynamically generated classes.  Please check the configuration for better
performance.*/

...I don't know how much this will weigh on performances.

Then I have a "B I N D I N G S  J O U R N A L" section with thousands of
lines like the followings:

/*operation@AddRecordTX;txID=109,recordID=110;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding
[id=110,
name=b7478faf-f0fd-481a-93ef-b6109dc3a689..cro.plantid.1073403.gwid.25324.device_profile.04.set,
address=.cro.plantid.1073403.gwid.25324.device_profile.04.set,
filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
(AMQAddress = 'activemq.notifications')), user=gateway, autoCreated=false]
operation@Commit;txID=109,numberOfRecords=1*/

...where "cro.plantid.1073403.gwid.25324.device_profile.04.set" is an
example of a topic for a specific plant (there are about 48 topic for
plant).
What does "isUpdate=false;compactCount=0;PersistentQueueBindingEncoding"
mean?

Then I have a "### Surviving Records Summary ###" section, again with
thousands of lines like this:

/*recordID=5205;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding
[id=5205, name=1447f6a659bc9bc0df2e55..cro.plantid.1071617.gwid.30889.
device_profile.02.err,
address=.cro.plantid.1071617.gwid.30889.device_profile.02.err,
filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
(AMQAddre
ss = 'activemq.notifications')), user=gateway, autoCreated=false]*/

Then I have the section "### Prepared TX ###", which is empty, and the ###
Message Counts ### as following:
message count=0
message reference count
prepared message count=0*/

Then I have the "M E S S A G E S   J O U R N A L" section with thousands of
lines like the followings:

peration@AddRecord;recordID=10088;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=10086;msg=ServerMessage[messageID=10086,durable=true,userID=nu
ll,priority=0, bodySize=512, timestamp=0,expiration=0, durable=true,
address=$sys.mqtt.retain..cro.plantid.1073620.gwid.32646.device_state.err,properties=TypedProp
erties[mqtt.message.retain=true,mqtt.qos.level=1]]@1144068272
operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=65]
operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=9747]
operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=7774]

... and no deleteRecord operation for that id (10088): I don't know if it is
right.

Then I have again the "### Surviving Records Summary ###" with thousands of
line like the followings;

recordID=196;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=196;msg=ServerMessage[messageID=196,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1073403.gwid.25324.device_profile.04,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@207430813
recordID=196;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=65]

...again my ignorance: why "durable=true"? because retained? because qos2?

for the MESSAGES JOURNAL I have the ### Message Counts ### with thousands of
lines like these:
message count=55958
message reference count
queue id 2,count=11
queue id 196608,count=2
queue id 4,count=1
queue id 196619,count=4
queue id 65547,count=1
queue id 196625,count=2
queue id 196630,count=2
queue id 65559,count=1
queue id 1081351,count=1
queue id 163869,count=17
queue id 163875,count=2307
... what does this mean?

Finally I have the "P A G I N G" section with thousands of lines like the
following:

/*Exploring store .cro.plantid.1074918.gwid.33300.device_profile.02 folder =
/app/mqtt/activeMQ/brokers/eltbroker01/./data/paging/5ac5a57d-dcc1-11e6-a52b-069133a4f5c
f
*******   Page 1
17:40:26,550 DEBUG [org.apache.activemq.artemis.core.paging.impl.Page]
reading page 1 on address =
.cro.plantid.1074918.gwid.33300.device_profile.02
pg=1, msg=0,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1034283,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@1029639001,Queues
= 1034178
pg=1, msg=1,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1036477,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@745530122,Queues
= 1036418,103417
8,1035409
pg=1, msg=2,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1038280,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@752575846,Queues
= 1036418,103797
9,1034178,1035409,1036769,1037082
...
*/

...but, in this case for example, since on topic/address
"cro.plantid.1074918.gwid.33300.device_profile.02" travel only retained
messages, I thought that at anytime there could be at most only 1 message
per time (because the next retained will substitute the previous one)...
It's not so?
And also here: why durable true? Because QoS2? Does "durable=true" mean that
these messages will persist in memory forever?

sorry for these patches of information ...but the result of "data print"
command was too big for an upload.
Anyway, hope can help you to help me.

Francesco


--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721395.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.