activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nielsbaloe <ni...@geoxplore.nl>
Subject Re: 100% CPU
Date Wed, 03 Apr 2013 17:42:35 GMT
Hi,

Thanks for your thoughts.

What do you mean with an activeMQ pool? Can you point out a reference?
And why do you think that with only a few events an hour the process would
go to 100% immediately after startup?

We'll check out the load without kahaDb, but note that the worker/consumer
machine does not even have a kahaDb instance, so I can't see why it should
matter for the consumer/worker.

For the other issues:
- Java 1.7.0_03 was simply the most recent when we installed everything,
same with the activeMQ 5.6 version itsself. I'll switch to the latest
activeMQ version, I hope you remember a bug in our activeMQ version that
causes this.
- you're right about the multiples for xmx and xms, but we have had
multiples before as well and then also had 100% load. The broker runs
within several meg anyway but has a 100% load.

Here is the top on the worker/consumer with 12GB memory:


~top - 19:45:21 up 30 days,  3:26,  1 user,  load average: 1.02, 1.04, 1.05
Tasks:  83 total,   1 running,  82 sleeping,   0 stopped,   0 zombie
Cpu(s): 17.0%us,  8.2%sy,  0.0%ni, 74.8%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Mem:  11899728k total,   725212k used, 11174516k free,   148112k buffers
Swap: 12056572k total,        0k used, 12056572k free,   320760k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  405 user      20   0 1736m 200m  10m S  100  1.7 214:55.31 java
  829 zabbix    20   0  8680 1160  752 S    0  0.0  35:27.80 zabbix_agentd
  830 zabbix    20   0  8680 1192  740 S    0  0.0  28:28.26 zabbix_agentd
  833 zabbix    20   0  8680 1192  740 S    0  0.0  29:18.04 zabbix_agentd
    1 root      20   0  3520 1836 1248 S    0  0.0   0:15.46 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.38 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:33.71 ksoftirqd/0
    5 root      20   0     0    0    0 S    0  0.0   0:00.52 kworker/u:0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S    0  0.0   0:07.38 watchdog/0
    8 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
    9 root      20   0     0    0    0 S    0  0.0   0:10.29 kworker/1:0
   10 root      20   0     0    0    0 S    0  0.0   0:02.46 ksoftirqd/1
   12 root      RT   0     0    0    0 S    0  0.0   0:05.08 watchdog/1
   13 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/2
   14 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/2:0
   15 root      20   0     0    0    0 S    0  0.0   0:05.08 ksoftirqd/2


Thanks!
Niels

>
>
> So, here're a few of my observations and suggestions based on the provided
> info:
> 1. use a newer java than v1.7.0_03
> 2. switch to amq v5.7 or v5.8
> 3. keep xmx and xms the same and preferably a multiple of 512 megs
> 4. where are you getting server grade machines with 2 and 15 gig ram
> 5. when you notice 100% util, take a few spaced apart thread dumps and
> post them on here
> 6. on a 4 core machine, load should not just be 100% but higher than 1.
> can you post 'top' output as well
> 7. use the amq pool for producers, sessions, connections where possible
> 8. test your existing config and load without kahadb (non-persistent mode)
> and compare results
>
>
> On Apr 3, 2013, at 7:09, nielsbaloe <niels@geoxplore.nl> wrote:
>
> Hi,
>
> -Xms512m -Xmx1500m
>
> but before we've set anything, it went wrong as well, and setting it
> higher also doesn't help. Note that there are only a few messages going
> through every hour, this is not yet fully production, so it should not
> even hit the 1% continuously...
>
> Thanks!,
> Niels
>
>>
>> Niels,
>>
>> I am not one of the experts here, but I am a new user of ActiveMQ via
>> TomEE, and I like to listen in on the mail topics/questions/responses
>> here.
>>
>> Since I listen in on Tomcat user list as well, I would say that this
>> sounds
>> like a GC (garbage collection) issue, but I might be mistaking.
>>
>> Can you reply with the java options of your app/container? What is your
>> java options for the Windows 7 laptops and the Ubuntu Linux
>> server/machine
>> that has 12GB and 2GB?
>>
>> I assume you have smaller memory settings on your Windows 7 (developer)
>> laptops and probably larger memory settings on the Ubuntu Linux
>> server/machine. Right? If yes, then GC may be the reason why you are
>> always
>> experiencing 100% CPU.
>>
>> This is just a guess/hunch, but since you provided such a detailed
>> question, please do not leave out the java (memory) options on your
>> Windows
>> 7 laptops as well as Ubuntu Linux server(s).
>>
>> Howard
>>
>>
>> On Wed, Apr 3, 2013 at 8:51 AM, nielsbaloe <niels@geoxplore.nl> wrote:
>>
>>> Hi all,
>>>
>>> We are using activeMQ successfully for two projects now, but we
>>> accidentely
>>> discovered that both the broker and the worker/consumer machines are
>>> hitting
>>> 100% CPU continuously. We do not have this issue on our developers
>>> machines
>>> (all Windows 7 laptops). This occurs even when no events are being
>>> processed.
>>>
>>> I couldn't find any clues for this issue, except setting the prefetch
>>> size.
>>> I've set the prefetch size to 10, as we have 10 consumers at the
>>> worker/consumer machine. We have a broker machine and a worker/consumer
>>> machine, which are both configured like below. In the near future we
>>> will
>>> add more worker/consumer machines.
>>>
>>> OS: Ubuntu Linux 12.04 (headless)
>>> Memory: 12GB and 2GB
>>> CPU: Intel Xeon 3.06GHz 4core
>>> Java: "1.7.0_03", OpenJDK Runtime Environment (IcedTea7 2.1.1pre)
>>> (7~u3-2.1.1~pre1-1ubuntu3)
>>> Webcontainer: none, java-standalone
>>> ActiveMQ: 5.6.0
>>>
>>> The broker uses the internal KahaDB database.
>>>
>>> We are using one queue, to which the worker/consumer machine is
>>> listening
>>> and posting to, say about 100 messages a day. We also use about 4
>>> scheduled
>>> messages for every 'modem' (our internal subject) which results in
>>> about
>>> 40
>>> scheduled messages or so which generates an event once every 30 minuts.
>>> Nothing spectacular so to say.
>>>
>>> Thanks for any clues in advance. For completeness, I will post our
>>> Broker,
>>> Consumer and Producer code (without comments), this might show any
>>> wrong
>>> assumptions on our side.
>>>
>>> Best,
>>> Niels Baloe
>>>
>>>
>>>> -----------------------
>>>
>>> public class Consumer implements ExceptionListener {
>>>
>>>        private Session session;
>>>        private MessageConsumer messageConsumer;
>>>        private static Logger LOG =
>>> Logger.getLogger(Consumer.class.getName());
>>>
>>>        public Consumer(String brokerServer, String queueName,
>>>                        MessageListener messageListener) throws
>>> JMSException,
>>>                        FileNotFoundException, IOException {
>>>                this(Broker.getSession(brokerServer), queueName,
>>> messageListener);
>>>        }
>>>
>>>        public Consumer(Session session, String queueName,
>>>                        MessageListener messageListener) throws
>>> JMSException {
>>>                this.session = session;
>>>
>>>                Queue queue = session.createQueue(queueName);
>>>                messageConsumer = session.createConsumer(queue);
>>>                messageConsumer.setMessageListener(messageListener);
>>>        }
>>>
>>>        public void close() {
>>>                try {
>>>                        messageConsumer.close();
>>>                } catch (JMSException e) {
>>>                }
>>>                try {
>>>                        session.close();
>>>                } catch (JMSException e) {
>>>                }
>>>        }
>>>
>>>        @Override
>>>        public void onException(JMSException je) {
>>>                LOG.log(Level.SEVERE, je.getMessage(), je);
>>>        }
>>>
>>> }
>>>
>>> public class Producer {
>>>
>>>        private Session session;
>>>        private MessageProducer producer;
>>>        public Producer(String brokerUrl, String queue) throws
>>> JMSException,
>>>                        FileNotFoundException, IOException {
>>>                this(Broker.getSession(brokerUrl), queue);
>>>        }
>>>
>>>        public Producer(Session session, String queue) throws
>>> JMSException
>>> {
>>>                this.session = session;
>>>                Destination destination = session.createQueue(queue);
>>>                producer = session.createProducer(destination);
>>>                producer.setDeliveryMode(DeliveryMode.PERSISTENT);
>>>        }
>>>
>>>        public void close() {
>>>                try {
>>>                        producer.close();
>>>                } catch (JMSException e) {
>>>                }
>>>                try {
>>>                        session.close();
>>>                } catch (JMSException je) {
>>>                }
>>>        }
>>>
>>>        public Message getMessageText(String text) throws JMSException {
>>>                return session.createTextMessage(text);
>>>        }
>>>
>>>        public Message getMessageObject() throws JMSException {
>>>                return session.createObjectMessage();
>>>        }
>>>
>>>        public void send(Message message) throws JMSException {
>>>                producer.send(message);
>>>        }
>>>
>>>        public void sendScheduled(Message message, String cron) throws
>>> JMSException
>>> {
>>>
>>> message.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_CRON, cron);
>>>                producer.send(message);
>>>        }
>>>
>>> }
>>>
>>> public class Broker {
>>>
>>>        private BrokerService broker;
>>>
>>>        public Broker(String host, int port, String brokerName) throws
>>> Exception {
>>>                broker = new BrokerService();
>>>                broker.setUseJmx(true);
>>>                broker.setBrokerName(brokerName);
>>>                broker.addConnector("tcp://" + host + ":" + port);
>>>                broker.setSchedulerSupport(true);
>>>                broker.start();
>>>        }
>>>
>>>        public URI getNameTCP() {
>>>                return broker.getVmConnectorURI();
>>>        }
>>>
>>>        public void close() {
>>>                try {
>>>                        broker.stop();
>>>                        broker.waitUntilStopped();
>>>                } catch (Exception e) {
>>>                }
>>>        }
>>>
>>>        public static void closeConnection() {
>>>                if (connection != null) {
>>>                        try {
>>>                                connection.close();
>>>                        } catch (JMSException e) {
>>>                        }
>>>                }
>>>        }
>>>
>>>        private static Connection connection;
>>>
>>>        private static Session getSessionWithoutRetry(String
>>> brokerServer)
>>>                        throws JMSException, FileNotFoundException,
>>> IOException {
>>>                if (connection == null) { // does not work when broker
>>> is
>>> local
>>>                        ActiveMQConnectionFactory connectionFactory =
>>> new
>>> ActiveMQConnectionFactory(
>>>                                        brokerServer);
>>>                        connectionFactory.setAlwaysSessionAsync(true);
>>>
>>>                        // Prefetch size
>>>                        String prefetch =
>>> NoImportUtils.getSettings().getProperty(
>>>                                        "broker.prefetchSize");
>>>
>>>                        connectionFactory.getPrefetchPolicy().setAll(
>>>                                        Integer.parseInt(prefetch));
>>>                        connection =
>>> connectionFactory.createConnection();
>>>                        connection.start();
>>>                }
>>>                return connection.createSession(false,
>>> Session.AUTO_ACKNOWLEDGE);
>>>        }
>>>
>>>        public static Session getSession(String brokerServer) throws
>>> JMSException,
>>>                        FileNotFoundException, IOException {
>>>                try {
>>>                        return getSessionWithoutRetry(brokerServer);
>>>                } catch (ConnectionFailedException e) {
>>>                        // Retry once when connection failed
>>>                        closeConnection();
>>>                        return getSessionWithoutRetry(brokerServer);
>>>                }
>>>        }
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>>
>> _______________________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665415.html
>>
>> To unsubscribe from 100% CPU, visit
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4665414&code=bmllbHNAZ2VveHBsb3JlLm5sfDQ2NjU0MTR8NTk0NzAwMTI4
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665420.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> _______________________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665431.html
>
> To unsubscribe from 100% CPU, visit
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4665414&code=bmllbHNAZ2VveHBsb3JlLm5sfDQ2NjU0MTR8NTk0NzAwMTI4






--
View this message in context: http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665442.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message