activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From magellings <mark.gelli...@qg.com>
Subject Re: Getting negative number of pending messages
Date Tue, 15 Sep 2009 17:50:15 GMT


I'm attaching a sample app, configurations, and logs in a zip file.  This
uses the official 1.1 NMS provider.  Ran a test against a 5.2 broker...first
starting producer sending 100 messages.  Shortly after started consumer
while those messages were still getting produced.  Everything went through
fine using ActiveMQ 5.2.  

Unzipped 5.3 snapshot (downloaded 9/9/09 8:41AM).  Changed tcp port to 10198
and jetty port to 10199 (as windows server 08 previously had firewall rules
set). All else in 5.3 unmodified.  Same test...it works...but get the
following error every so often...

Failed to register MBean

Then I added in jdbc master/slave (config in zip file).  I placed the sql
and c3p0 jar files in \lib\optional as necessary.

Producer gets to about message 13 and then it freezes.  Consumer too.  

Broker reports the following errors...

Failed to register MBean

Ignoring ack received before dispatch; result of failover with an
outstanding ack. Acked messages will be replayed if present on this broker.
Ignored ack: MessageAck {commandId = 5, responseRequired = false, ackType =
2, consumerId = 48c400d4-2192-4e3e-8adc-aece3f27803f:1:1, firstMessageId =
08b5f3e4-a931-49f5-99cc-aeb1cdd313d6:1:1:1, lastMessageId =
08b5f3e4-a931-49f5-99cc-aeb1cdd313d6:1:1:1, destination = queue://test3,
transactionId = TX:48c400d4-2192-4e3e-8adc-aece3f27803f:1, messageCount = 1}

And then trying to shutdown the broker (Ctrl C) I get this of which IP is my
machine.  Broker runs on a Windows Server 2008 machine.

The connection to '/10.0.1.5:2150' is taking a long time to shutdown.

At which point I need to kill the broker.

I wasn't able to reproduce the previous "Async error occurred" exceptions. 
Perhaps migrating to official NMS 1.1 helped.

If someone with a bit more NMS experience and 5.3 experience could overlook
this that would be great.  Seems like something with jdbc master/slave may
be messed up.

If I get some help hopefully I can verify the original problem (caching bug)
is fixed in 5.3 as it is suppose to be.


magellings wrote:
> 
> I'm working on getting upgraded to official 1.1.0.1642 version right now. 
> Previously I think I was using a release candidate.  This requires
> rebuilding our framework that sits on top of the NMS provider.  Running my
> unit tests against "5.2" (for sanity check) as we speak.  I'll then
> perform the test again against 5.3 with the official 1.1.
> 
> 
> Timothy Bish wrote:
>> 
>> 
>> What version of the NMS client are you using?  Do you have a sample app
>> that can causes this to happen I'd like to try and reproduce the NMS
>> errors if possible.
>> 
>> Regards
>> Tim.
>> 
>> 
>> magellings wrote:
>>> 
>>> Note test went from 2009-09-09 14:01:29,918 to about 14:06.
>>> 
>>> I also needed to kill the broker.
>>> 
>>> 
>>> magellings wrote:
>>>> 
>>>> Using the wild-card address allowed the producers to connect.
>>>> 
>>>> I'm getting a lot of exceptions in the activemq.log however with 5.3 so
>>>> I can't test this.  It is very slow too.  I guess this isn't surprising
>>>> if exceptions are occurring.
>>>> 
>>>> I uploaded the activemq.log and a detailed NMS producer/consumer log. 
>>>> Possibly this is something with the NMS provider?
>>>> 
>>>>  http://www.nabble.com/file/p25371640/log.txt consumerLog.txt 
>>>>  http://www.nabble.com/file/p25371640/Log.txt producerLog.txt  
>>>>  http://www.nabble.com/file/p25371640/activemq.log activemq.log 
>>>> 
>>>> 
>>>> Gary Tully wrote:
>>>>> 
>>>>> with 5.3 "localhost" is local, not the wildcard address. you may need
>>>>> to use
>>>>> the wildcard address "0.0.0.0" if you localhost is tied to a different
>>>>> interface.
>>>>> 
>>>>> 2009/9/9 magellings <mark.gellings@qg.com>
>>>>> 
>>>>>>
>>>>>> Okay I have 5.3 configured with the following config.  I can't
>>>>>> connect from
>>>>>> another machine through the host name of the server however.  This
>>>>>> config
>>>>>> is
>>>>>> taken directly from an ActiveMQ 5.2 broker and running that broker
I
>>>>>> can
>>>>>> connect just fine.  Anyone know what's up?
>>>>>>
>>>>>> <!--
>>>>>>    Licensed to the Apache Software Foundation (ASF) under one or
more
>>>>>>    contributor license agreements.  See the NOTICE file distributed
>>>>>> with
>>>>>>    this work for additional information regarding copyright
>>>>>> ownership.
>>>>>>    The ASF licenses this file to You under the Apache License,
>>>>>> Version 2.0
>>>>>>    (the "License"); you may not use this file except in compliance
>>>>>> with
>>>>>>    the License.  You may obtain a copy of the License at
>>>>>>
>>>>>>    http://www.apache.org/licenses/LICENSE-2.0
>>>>>>
>>>>>>    Unless required by applicable law or agreed to in writing,
>>>>>> software
>>>>>>    distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>> implied.
>>>>>>    See the License for the specific language governing permissions
>>>>>> and
>>>>>>    limitations under the License.
>>>>>> -->
>>>>>>
>>>>>> <beans
>>>>>>  xmlns="http://www.springframework.org/schema/beans"
>>>>>>  xmlns:amq="http://activemq.apache.org/schema/core"
>>>>>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>  xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>>>>  http://activemq.apache.org/schema/core
>>>>>> http://activemq.apache.org/schema/core/activemq-core.xsd
>>>>>>  http://activemq.apache.org/camel/schema/spring
>>>>>> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>>>>>>
>>>>>>  <!-- Allows us to use system properties as variables in this
>>>>>> configuration
>>>>>> file -->
>>>>>>  <bean
>>>>>>
>>>>>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>>>>>    <property name="locations">
>>>>>>     
>>>>>> <value>file:///${activemq.base}/conf/credentials.properties</value>
>>>>>>    </property>
>>>>>>  </bean>
>>>>>>
>>>>>>  <broker xmlns="http://activemq.apache.org/schema/core"
>>>>>> brokerName="WAMQDEV1PRODUCTREQUEST"
>>>>>> dataDirectory="${activemq.base}/data">
>>>>>>
>>>>>>    <plugins>
>>>>>>      <simpleAuthenticationPlugin>
>>>>>>        <users>
>>>>>>          <authenticationUser username="GL$ACTIVEMQ_PRODUCTREQUEST"
>>>>>> password="replenish" groups="producers,consumers,admins" />
>>>>>>        </users>
>>>>>>      </simpleAuthenticationPlugin>
>>>>>>      <authorizationPlugin>
>>>>>>        <map>
>>>>>>          <authorizationMap>
>>>>>>            <authorizationEntries>
>>>>>>              <authorizationEntry queue=">" write="producers"
>>>>>> read="consumers" admin="admins,producers" />
>>>>>>              <authorizationEntry topic=">" read="admins"
>>>>>> write="admins,producers" admin="admins,producers,consumers" />
>>>>>>            </authorizationEntries>
>>>>>>          </authorizationMap>
>>>>>>        </map>
>>>>>>      </authorizationPlugin>
>>>>>>    </plugins>
>>>>>>
>>>>>>
>>>>>>    <!-- Destination specific policies using destination names
or
>>>>>> wildcards
>>>>>> -->
>>>>>>    <destinationPolicy>
>>>>>>      <policyMap>
>>>>>>        <policyEntries>
>>>>>>          <policyEntry queue=">" useCache="true" memoryLimit="5mb"/>
>>>>>>          <policyEntry topic=">" advisoryForSlowConsumers="true"
>>>>>> advisdoryForFastProducers="true" advisoryWhenFull="true"
>>>>>> memoryLimit="5mb"/>
>>>>>>        </policyEntries>
>>>>>>      </policyMap>
>>>>>>    </destinationPolicy>
>>>>>>
>>>>>>    <!-- Use the following to configure how ActiveMQ is exposed
in JMX
>>>>>> -->
>>>>>>    <managementContext>
>>>>>>      <managementContext createConnector="false"/>
>>>>>>    </managementContext>
>>>>>>
>>>>>>    <persistenceAdapter>
>>>>>>      <amqPersistenceAdapter syncOnWrite="false"
>>>>>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>>>>>>    </persistenceAdapter>
>>>>>>
>>>>>>    <persistenceAdapter>
>>>>>>      <jdbcPersistenceAdapter dataSource="#mssql-ds"
>>>>>> createTablesOnStartup="false"  />
>>>>>>    </persistenceAdapter>
>>>>>>
>>>>>>    <sslContext>
>>>>>>      <sslContext keyStore="file:${activemq.base}/conf/broker.ks"
>>>>>> keyStorePassword="password"
>>>>>> trustStore="file:${activemq.base}/conf/broker.ts"
>>>>>> trustStorePassword="password"/>
>>>>>>    </sslContext>
>>>>>>
>>>>>>    <!--  The maximum about of space the broker will use before
>>>>>> slowing down
>>>>>> producers -->
>>>>>>    <systemUsage>
>>>>>>      <systemUsage>
>>>>>>        <memoryUsage>
>>>>>>          <memoryUsage limit="20 mb"/>
>>>>>>        </memoryUsage>
>>>>>>        <storeUsage>
>>>>>>          <storeUsage limit="1 gb" name="foo"/>
>>>>>>        </storeUsage>
>>>>>>        <tempUsage>
>>>>>>          <tempUsage limit="100 mb"/>
>>>>>>        </tempUsage>
>>>>>>      </systemUsage>
>>>>>>    </systemUsage>
>>>>>>
>>>>>>    <!-- The transport connectors ActiveMQ will listen to -->
>>>>>>    <transportConnectors>
>>>>>>      <transportConnector name="openwire" uri="tcp://localhost:10198"
>>>>>> discoveryUri="multicast://default"/>
>>>>>>    </transportConnectors>
>>>>>>
>>>>>>  </broker>
>>>>>>
>>>>>>  <bean id="mssql-ds"
>>>>>> class="com.mchange.v2.c3p0.ComboPooledDataSource"
>>>>>> destroy-method="close">
>>>>>>    <property name="driverClass"
>>>>>> value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
>>>>>>    <property name="jdbcUrl"
>>>>>> value="jdbc:sqlserver://CHALLENGER.qg.com:1433
>>>>>> ;databaseName=ActiveMQProductRequest;user=GL$ACTIVEMQ_PRODUCTREQUEST;password=replenish"/>
>>>>>>    <property name="user" value="GL$ACTIVEMQ_PRODUCTREQUEST"/>
>>>>>>    <property name="password" value="replenish"/>
>>>>>>    <property name="acquireRetryAttempts" value="100"/>
>>>>>>    <property name="testConnectionOnCheckout" value="true"/>
>>>>>>    <property name="testConnectionOnCheckin" value="true"/>
>>>>>>    <property name="automaticTestTable" value="ACTIVEMQ_CONN_TEST"/>
>>>>>>    <property name="acquireRetryDelay" value="5000"/>
>>>>>>    <property name="preferredTestQuery" value="SELECT 1"/>
>>>>>>  </bean>
>>>>>>
>>>>>>  <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>>>>>>    <connectors>
>>>>>>      <nioConnector port="10199"/>
>>>>>>    </connectors>
>>>>>>
>>>>>>    <handlers>
>>>>>>      <webAppContext contextPath="/admin"
>>>>>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>>>>>>      <webAppContext contextPath="/camel"
>>>>>> resourceBase="${activemq.base}/webapps/camel" logUrlOnStart="true"/>
>>>>>>      <webAppContext contextPath="/demo"
>>>>>> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
>>>>>>      <webAppContext contextPath="/fileserver"
>>>>>> resourceBase="${activemq.base}/webapps/fileserver"
>>>>>> logUrlOnStart="true"/>
>>>>>>    </handlers>
>>>>>>  </jetty>
>>>>>>
>>>>>> </beans>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> rajdavies wrote:
>>>>>> >
>>>>>> > We are aiming to get a 5.3 release candidate to vote on next
week -
>>>>>> so
>>>>>> > no long
>>>>>> > On 4 Sep 2009, at 19:46, magellings wrote:
>>>>>> >
>>>>>> >>
>>>>>> >> Right now we're focusing pretty hard on stress testing version
5.2
>>>>>> >> but we can
>>>>>> >> try and fit in a test.  When is the estimated release date
of
>>>>>> >> version 5.3?
>>>>>> >> In other words, how much time do we have?  :)
>>>>>> >>
>>>>>> >>
>>>>>> >> rajdavies wrote:
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On 4 Sep 2009, at 15:14, magellings wrote:
>>>>>> >>>
>>>>>> >>>>
>>>>>> >>>> We too were having problems were number of pending
of messages
>>>>>> was a
>>>>>> >>>> negative
>>>>>> >>>> number and messages received was greater than number
sent.  We
>>>>>> >>>> determined
>>>>>> >>>> there were indeed duplicate messages being sent.
>>>>>> >>>>
>>>>>> >>>> We ended up setting useCache to false on the queue
policy entry
>>>>>> and
>>>>>> >>>> the
>>>>>> >>>> problem appears to be resolved.
>>>>>> >>>>
>>>>>> >>>> We too were sending a high volume of messages to
our queues
>>>>>> >>>> relatively
>>>>>> >>>> quickly.
>>>>>> >>>> --
>>>>>> >>>> View this message in context:
>>>>>> >>>>
>>>>>> http://www.nabble.com/Getting-negative-number-of-pending-messages-tp22106707p25295148.html
>>>>>> >>>> Sent from the ActiveMQ - User mailing list archive
at
>>>>>> Nabble.com.
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> It would be great to know if you still had the problem
on the
>>>>>> latest
>>>>>> >>> 5.3 snapshot too
>>>>>> >>>
>>>>>> >>> cheers,
>>>>>> >>>
>>>>>> >>> Rob
>>>>>> >>>
>>>>>> >>> Rob Davies
>>>>>> >>> I work here: http://fusesource.com
>>>>>> >>> My Blog: http://rajdavies.blogspot.com/
>>>>>> >>> I'm writing this: http://www.manning.com/snyder/
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>> >> --
>>>>>> >> View this message in context:
>>>>>> >>
>>>>>> http://www.nabble.com/Getting-negative-number-of-pending-messages-tp22106707p25299721.html
>>>>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>> >>
>>>>>> >
>>>>>> > Rob Davies
>>>>>> > I work here: http://fusesource.com
>>>>>> > My Blog: http://rajdavies.blogspot.com/
>>>>>> > I'm writing this: http://www.manning.com/snyder/
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Getting-negative-number-of-pending-messages-tp22106707p25369349.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>> 
>>>>> 
>>>>> -- 
>>>>> http://blog.garytully.com
>>>>> 
>>>>> Open Source Integration
>>>>> http://fusesource.com
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Getting-negative-number-of-pending-messages-tp22106707p25458685.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message