activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: ActiveMQ stops delivering messages to consumers when saturating a high-latency network link
Date Wed, 01 Oct 2014 16:10:31 GMT
Are you saying that the manual solution (a configurable threshold on a
broker's network connection to another broker) doesn't sound reasonable, or
just that the automatic solution (where the broker would measure end-to-end
latency all the way to the consumer) doesn't?  I agree that the automatic
solution is a) hard, and b) not in line with ActiveMQ's current approach of
brokers being agnostic of the full route to a consumer (though an exception
is made for decreaseNetworkConsumerPriority) so I don't expect it to be
implemented anytime soon.

But I think that the manual configuration option would provide a reasonable
but crude tool for dealing with high-latency networks, and it wouldn't
violate the JMS ordering guarantees since messages would be discarded or
not but never delivered out of order.

The problem set is pretty straight-forward: we're given a pretty limited
bandwidth allocation across a WAN, and although we've tried to estimate our
bandwidth usage to make sure we're going to stay within it, fluctuations in
usage patterns might push us beyond our allocation.  If that happens, we'd
like to continue processing in a degraded manner - some messages would be
lost and all messages would be processed just before their message
expiration time - rather than stop processing entirely, which is what
happens now.  So yes, ActiveMQ would be used here to throttle traffic
instead of allowing the network link to do so, to provide better end-to-end
functionality across the network of brokers as a whole.  And although
there's an obvious need for all ActiveMQ users to allocate adequate
bandwidth for their needs, having this option in ActiveMQ would provide a
safeguard for what happens if either the actual ActiveMQ usage or the
available bandwidth changes unexpectedly.

Your suggestion of adding jitter to the message expiration time is an
interesting one that I hadn't considered.  Since we're using Camel to send
our messages and to set that expiration time, it would be non-trivial to
implement, but it might give us a way to handle this if ActiveMQ doesn't.
But I think having a single setting in the config of each ActiveMQ
broker would be preferable to having to implement a way to use a jittery
expiration date and then configure that jitter on every producer we
configure, since it would reduce the odds of someone misconfiguring a
producer and not getting protected.

Tim

On Sun, Sep 28, 2014 at 12:25 AM, artnaseef <art@artnaseef.com> wrote:

> Honestly, that doesn't sound reasonable to me.
>
> ActiveMQ has no solid way to know which messages *should* or *should not*
> reach their destination before the TTL expires.  That would require some
> feedback from across the broker-to-broker connection so the sending broker
> could take that into account.  In addition, JMS has ordering guarantees
> (although it's never wise to fully depend on them).
>
> The problem set sounds interesting - almost like ActiveMQ is being used as
> a
> throttle for network traffic.
>
> Anyway, how about altering the message TTLs plus or minus the maximum
> expected network latency times two?  Either using randomness or simply
> alternating between + and -.  In that way, some messages will still survive
> the trip while others don't when the backlog of messages reaches that
> critical point.  And, some (most?) should expire before hitting the wire,
> saving that wasted trip.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-stops-delivering-messages-to-consumers-when-saturating-a-high-latency-network-link-tp4685557p4685951.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message