activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miroslav Novak (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ARTEMIS-921) Consumers killed as slow even if overall consuming rate is above threshold
Date Wed, 01 Mar 2017 15:06:45 GMT

    [ https://issues.apache.org/jira/browse/ARTEMIS-921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890330#comment-15890330
] 

Miroslav Novak commented on ARTEMIS-921:
----------------------------------------

There is still a problem how producer msg/s rate is calculated in {{QueueImpl.getRate()}}
for slow consumer detection. It calculates only messages added during the last {{queueRateCheckTime}}
seconds. As this is used to figure out, in which maximum msg/s rate the queue could serve
the consumer then it should also take into account messages which are already in queue at
{{queueRateCheckTime}} time. 

Current implementation is problem for cases when messages are sent to queue in bursts, for
example producer sends 1000s messages in a few seconds and then stops and will do that again
in 1 hour. QueueImpl.getRate() method returns 0 msg/s for queueRateCheckTime which is by default
5 seconds and slow consumer detection will be skipped (almost) all the time.

I tried to fix it by following change to QueueImpl.getRate() method and seems to be ok, wdyt?
{code}
   private final AtomicLong messageCountSnapshot = new AtomicLong(0);
   public float getRate() {
      long locaMessageAdded = getMessagesAdded();
      float timeSlice = ((System.currentTimeMillis() - queueRateCheckTime.getAndSet(System.currentTimeMillis()))
/ 1000.0f);
      if (timeSlice == 0) {
         messagesAddedSnapshot.getAndSet(locaMessageAdded);
         return 0.0f;
      }
      return BigDecimal.valueOf(((locaMessageAdded - messagesAddedSnapshot.getAndSet(locaMessageAdded))
+ messageCountSnapshot.getAndSet(getMessageCount())) / timeSlice).setScale(2, BigDecimal.ROUND_UP).floatValue();
<-- here is the change
   }
{code}

> Consumers killed as slow even if overall consuming rate is above threshold
> --------------------------------------------------------------------------
>
>                 Key: ARTEMIS-921
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-921
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 1.5.1
>            Reporter: Howard Gao
>            Assignee: clebert suconic
>             Fix For: 1.5.2, 2.0.0
>
>
> We have one queue. Imagine messages are produced at 2 msgs/s. There are three consumers
and slow consumer limit is 1 msgs/s. What happens is that all three consumers get killed as
slow, even though it is impossible for any of them to be fast, since messages are distributed
equally between the consumers (round-robin).
> This has real consumer impact in a situation when producer rate is usually high (so it
requires multiple consumers working in parallel), but may occasionally drop close to consumer-threshold.
In this case, broker disconnects all consumers who then have to reconnect and message processing
is delayed for the time of the reconnecting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message