activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sthan001 <tsaravananthanga...@gmail.com>
Subject ActiveMQ Keeps tmp db-log files on the name of old/disconnected Topic Subscription Connection ID
Date Mon, 08 Feb 2016 11:32:39 GMT
Hi AMQ Developer/User,Noticed below scenario by active mq on the heavy
message processing.Active MQ Directly routes new messages to tmp_storage if
broker/queue size reach 70% of memory configured. On heavy messages arrival,
the no of db log files increasing.Consumer to an queue reads messages and
broadcasts via topic. We had very less memory for broker, so we go topic
subscription full for the topic destination. We increased broker size from
default 128MB to several GB to hold atleast 2L messages on memory.When there
are several clients to same topic destination around 20 [slow network
connection + rate is 50events dispatch per second from queue to topic - so
20*50 events dispacth to be made by topic], we suspect we get file
descriptors misbehavior on tmp_storage of active mq with referencing a
disconnected/died topic consumer name.Below is sample db-log files to be
cleard by active mq scheduler. This 1099 file is kept not clearing for more
than 2 hours. This causing the topic publisher to push a message in a very
delayed manner.2016-02-08 17:09:10,640 [ActiveMQ Broker[localhost]
Scheduler] TRACE: plist.PListStoreImpl - Remaining gc candidate set after
refs from:
800_TopicSubscription:786[ID:kowalski-42321-1454875772406-5:1:1:1]:[1099,
1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111,
1112, 1113, 1114, 1115, 1116, 1117, 1118, 1119, 1120, 1121, 1122, 1123,
1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135,
1136, 1137, 1138, 1139, 1140, 1141, 1142, 1143, 1144, 1145, 1146, 1147,
1148, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159,
1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 1170, 1171,
1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 1182, 1183,
1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195,
1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1205, 1206, 1207,
1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219,
1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230, 1231,
1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241, 1242, 1243,
1244, 1245, 1246, 1247, 1248, 1249, 1250, 1251, 1252, 1253, 1254, 1255,
1256, 1257, 1258, 1259, 1260, 1261, 1262, 1263, 1264, 1265, 1266, 1267,
1268, 1269, 1270, 1271, 1272, 1273, 1274, 1275, 1276, 1277, 1278, 1279,
1280, 1281, 1282, 1283, 1284, 1285, 1286, 1287, 1288, 1289, 1290, 1291,
1292, 1293, 1294, 1295, 1296, 1297, 1298, 1299, 1300, 1301, 1302, 1303,
1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315,
1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327,
1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 1338, 1339,
1340, 1341, 1342, 1343, 1344, 1345, 1346, 1347, 1348, 1349, 1350, 1351,
1352, 1353, 1354, 1355, 1356, 1357, 1358, 1359, 1360, 1361, 1362, 1363,
1364, 1365, 1366, 1367, 1368, 1369, 1370, 1371, 1372, 1373, 1374, 1375,
1376, 1377, 1378, 1379, 1380, 1381, 1382, 1383, 1384, 1385, 1386, 1387,
1388, 1389, 1390, 1391, 1392, 1393, 1394, 1395, 1396, 1397, 1398, 1399,
1400, 1401, 1402, 1403, 1404, 1405, 1406, 1407, 1408, 1409, 1410, 1411,
1412, 1413, 1414, 1415, 1416, 1417, 1418, 1419, 1420, 1421, 1422, 1423,
1424, 1425, 1426, 1427, 1428, 1429, 1430, 1431, 1432, 1433, 1434, 1435,
1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 1446, 1447,
1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459,
1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471,
1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1481, 1482, 1483,
1484, 1485, 1486, 1487, 1488, 1489, 1490, 1491, 1492, 1493, 1494, 1495,
1496, 1497, 1498, 1499, 1500, 1501, 1502, 1503, 1504, 1505, 1506, 1507,
1508, 1509, 1510, 1511, 1512, 1513, 1514, 1515, 1516, 1517, 1518, 1519,
1520, 1521, 1522, 1523, 1524, 1525, 1526, 1527, 1528, 1529, 1530, 1531,
1532, 1533, 1534, 1535, 1536, 1537, 1538, 1539, 1540, 1541, 1542, 1543,
1544, 1545, 1546, 1547, 1548, 1549, 1550, 1551, 1552, 1553, 1554, 1555,
1556, 1557, 1558, 1559, 1560, 1561, 1562, 1563, 1564, 1565, 1566, 1567,
1568, 1569, 1570, 1571, 1572, 1573, 1574, 1575, 1576, 1577, 1578, 1579,
1580, 1581, 1582, 1583, 1584, 1585, 1586, 1587, 1588, 1589, 1590, 1591,
1592, 1593, 1594, 1595, 1596, 1597, 1598, 1599, 1600, 1601, 1602, 1603,
1604, 1605, 1606, 1612, 1636, 1637, 1638, 1639, 1640, 1641, 1642, 1643,
1644, 1645, 1646, 1647, 1648, 1649, 1650, 1651, 1652, 1653, 1654, 1655,
1656, 1657, 1658, 1659, 1660, 1661, 1662, 1663, 1664, 1665, 1666, 1667,
1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679,
1680, 1681, 1682, 1683, 1684, 1685, 1686, 1687, 1688, 1689, 1690, 1691,
1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703,
1704, 1705, 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715,
1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 1726, 1727]We
got below stack trace at Topic Publisher:"ActiveMQ Session Task-115" prio=10
tid=0x00002aaac42a0800 nid=0x2f44 waiting for monitor entry
[0x000000005d8be000]   java.lang.Thread.State: BLOCKED (on object monitor)       
at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1737)       
- waiting to lock <0x000000062a474e68> (a java.lang.Object)        at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:289)       
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:224)       
at
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
      
at
org.apache.activemq.ActiveMQTopicPublisher.publish(ActiveMQTopicPublisher.java:123)We
got Active MQ size Transport Thread BLOCK:"ActiveMQ Transport:
tcp:///192.168.241.220:57907@61616" daemon prio=10 tid=0x0000000008f02000
nid=0x24f1 waiting for monitor entry [0x0000000044108000]  
java.lang.Thread.State: BLOCKED (on object monitor)        at
org.apache.activemq.store.kahadb.plist.PListImpl$PListIteratorImpl.(PListImpl.java:232)  
    
- waiting to lock <0x00000007000cc268> (a java.lang.Object)        at
org.apache.activemq.store.kahadb.plist.PListImpl.iterator(PListImpl.java:222)       
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.(FilePendingMessageCursor.java:499)
      
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.pageInList(FilePendingMessageCursor.java:184)
      
- locked <0x0000000704900ce8> (a
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)       
at
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:174)      

- locked <0x0000000704900ce8> (a
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)        -
locked <0x0000000704904560> (a java.lang.Object)        at
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:48)
      
at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:688)       
at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:499)       
- locked <0x00000007000ea218> (a org.apache.activemq.broker.region.Topic)       
at org.apache.activemq.broker.region.Topic.send(Topic.java:435)        at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:406)       
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:392)       
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:282)    
  
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)       
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
      
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:317)       
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)       
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:499)
      
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:749)       
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)     
 
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184) 
     
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)       
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
      
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
      
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)       
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)       
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)       
at
java.lang.Thread.run(Thread.java:724)====================================Any
Suggestion is much appreciated.Suspects & Questions:1. if old/dead topic is
referenced for longer time, would the heap/ram filled with messages which
can be discarded and can't be cleaned while garbage collection. So 40% heap
filled with these kind of messages and new messages will be directed to file
instead of RAM?2. How frequent the topic connection is monitored and it's
messages are discarded from ram as well as from file?Thanks,Saravanan
Thangavel



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Keeps-tmp-db-log-files-on-the-name-of-old-disconnected-Topic-Subscription-Connection-ID-tp4707113.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message