activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: ActiveMQ should GC inactive when producers is 0 , even if consumers is > 0
Date Mon, 02 Feb 2015 20:23:06 GMT
FYI.. I closed this bug as WONTFIX as I think this is the best strategy
post feedback.

Thanks guys.

On Mon, Feb 2, 2015 at 11:52 AM, Kevin Burton <burton@spinn3r.com> wrote:

> the problem with this strategy is that queues can come and go.. it’s a
> scheduler system for a web crawler. so if a URL request comes in for an IP
> we place it in that queue.  And if, a month from now, we have to request
> another resources from that IP, we place it in a queue for that IP.
>
> MOST of the time, we just have a few IPs because most web properties are
> large. However, sometimes, we have a request once per hour to an IP.  In
> that situation we want to GC the queue 60 seconds later.
>
> Kevin
>
> On Mon, Feb 2, 2015 at 11:17 AM, Tim Bain <tbain@alumni.duke.edu> wrote:
>
>> It sounds like the advisory message you mentioned should let you work
>> around the problem.
>>
>> But what it sounds like you really need is a way to know, for sure, that
>> the producer is finished using a queue.  If your producers write an EOF
>> message when they're through, your consumer can then safely go away
>> without
>> having to worry about stragglers.  Then your only remaining problem is how
>> to know about producers that die unexpectedly without writing the EOF or
>> that take a really long time to producer their next message.  A heartbeat
>> mechanism would solve those issues: every X amount of time, the producer
>> produces a small no-op heartbeat message that tells the consumer it's
>> still
>> there; failure to receive one of those messages after N heartbeat
>> intervals
>> indicates that the producer is no longer alive.  The only thing it doesn't
>> protect you against is temporary network perturbations (since they would
>> make it appear that the producer is gone when it's not really), but you
>> might be able to live with that.
>>
>> I'm not sure that heartbeats are a better solution than processing the
>> advisory messages as you've suggested, but I think at a minimum you should
>> have an EOF that's always sent (during normal operations) when the
>> producer
>> closes, and to which your consumer reacts by disconnecting as well.  Then
>> the advisory messages can be used only for handling truly exceptional
>> conditions.
>>
>> On Mon, Feb 2, 2015 at 11:29 AM, Kevin Burton <burton@spinn3r.com> wrote:
>>
>> > > ActiveMQ provides no means to guarantee that a producer will fail to
>> > > deliver
>> > > a message when there are no consumers interested in the message.  It
>> is
>> > > actually designed more for cases of consumers being away and returning
>> > at a
>> > > later time.
>> > >
>> > >
>> > Perhaps. But I think the advisory queue message
>> >
>> > ActiveMQ.Advisory.NoConsumer.Queue
>> >
>> > … actually solves my problem.
>> >
>> > If a producer creates a queue, and no one is consuming, I’ll just
>> create a
>> > new consumer once I get the advisory message about the queue.
>> >
>> > This is actually how I do it *now* except I was listening to NEW queues…
>> >
>> > I’ve actually gotten a lot of use out of the advisory queue system.
>> >
>> >
>> > > As you noted correctly, ActiveMQ automatically recreates destinations
>> > every
>> > > time they are needed, so even if they were GC'ed, they would just be
>> > > automatically recreated when a producer posted a message.
>> > >
>> >
>> > Yes.  And in my case, I think that’s ok.
>> >
>> > I might actually write up a blog post about this design. It’s novel but
>> AMQ
>> > actually does seem to be working out well for us without the limitations
>> > you describe due to advisory messages.
>> >
>> > Kevin
>> >
>> > --
>> >
>> > Founder/CEO Spinn3r.com
>> > Location: *San Francisco, CA*
>> > blog: http://burtonator.wordpress.com
>> > … or check out my Google+ profile
>> > <https://plus.google.com/102718274791889610666/posts>
>> > <http://spinn3r.com>
>> >
>>
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>


-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message