activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Swapnonil Mukherjee (JIRA)" <j...@apache.org>
Subject [jira] Updated: (AMQ-3103) Active MQ stalls after Job Scheduler component shuts down.
Date Thu, 23 Dec 2010 10:50:00 GMT

     [ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------

    Description: 
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the
scheduler component processing scheduled messages on that queue encounters an Null Pointer
Exception.

Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place
a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}

We use the Spring Default Message Listener Container to receive messages.

Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on
the Active MQ Console, after which they processed normally and we can tally using the "Messages
Enqueued" and  the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with
the following exception.

{code:xml}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl
| JobScheduler:JMS
java.lang.NullPointerException
        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
        at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
        at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
        at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
        at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO  | JobSchedulerStore:activemq-data/primary/scheduler stopped
| org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}

Why does the Job Scheduler fail? One possible reason we have found is that the clock time
settings on the VMs producing the messages and the broker as well as the consumers are all
different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference
in clock. We are in the process of syncing all clocks but we are not sure whether this will
solve the problem.

Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it
shutdown? Even more problematic is the fact that the queue itself stalls and does not accept
anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker
recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down
were lost. We have not been able to recover those messages.

I am attaching the activemq log and activemq configuration file.




  was:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the
scheduler component processing scheduled messages on that queue encounters an Null Pointer
Exception.

Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place
a delay of 30 seconds on each message before posting
{code:xml} 
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}

We use the Spring Default Message Listener Container to receive messages.

Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on
the Active MQ Console, after which they processed normally and we can tally using the "Messages
Enqueued" and  the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with
the following exception.

{noformat}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl
| JobScheduler:JMS
java.lang.NullPointerException
        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
        at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
        at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
        at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
        at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
        at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO  | JobSchedulerStore:activemq-data/primary/scheduler stopped
| org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}

Why does the Job Scheduler fail? One possible reason we have found is that the clock time
settings on the VMs producing the messages and the broker as well as the consumers are all
different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference
in clock. We are in the process of syncing all clocks but we are not sure whether this will
solve the problem.

Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it
shutdown? Even more problematic is the fact that the queue itself stalls and does not accept
anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker
recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down
were lost. We have not been able to recover those messages.

I am attaching the activemq log and activemq configuration file.





> Active MQ stalls after Job Scheduler component shuts down.
> ----------------------------------------------------------
>
>                 Key: AMQ-3103
>                 URL: https://issues.apache.org/jira/browse/AMQ-3103
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: Redhat Enterprise Linux 5.X, 
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Active MQ 4.2
>            Reporter: Swapnonil Mukherjee
>         Attachments: activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after
the scheduler component processing scheduled messages on that queue encounters an Null Pointer
Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also
place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30")
* 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled"
tab on the Active MQ Console, after which they processed normally and we can tally using the
"Messages Enqueued" and  the "Messages Dequeued" numbers. But occasionally the Job Scheduler
fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl
| JobScheduler:JMS
> java.lang.NullPointerException
>         at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
>         at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
>         at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
>         at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
>         at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
>         at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>         at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
>         at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
>         at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
>         at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO  | JobSchedulerStore:activemq-data/primary/scheduler stopped
| org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock
time settings on the VMs producing the messages and the broker as well as the consumers are
all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference
in clock. We are in the process of syncing all clocks but we are not sure whether this will
solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should
it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept
anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker
recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down
were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message