activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: ActiveMQ should GC inactive when producers is 0 , even if consumers is > 0
Date Mon, 02 Feb 2015 19:52:08 GMT
the problem with this strategy is that queues can come and go.. it’s a
scheduler system for a web crawler. so if a URL request comes in for an IP
we place it in that queue.  And if, a month from now, we have to request
another resources from that IP, we place it in a queue for that IP.

MOST of the time, we just have a few IPs because most web properties are
large. However, sometimes, we have a request once per hour to an IP.  In
that situation we want to GC the queue 60 seconds later.

Kevin

On Mon, Feb 2, 2015 at 11:17 AM, Tim Bain <tbain@alumni.duke.edu> wrote:

> It sounds like the advisory message you mentioned should let you work
> around the problem.
>
> But what it sounds like you really need is a way to know, for sure, that
> the producer is finished using a queue.  If your producers write an EOF
> message when they're through, your consumer can then safely go away without
> having to worry about stragglers.  Then your only remaining problem is how
> to know about producers that die unexpectedly without writing the EOF or
> that take a really long time to producer their next message.  A heartbeat
> mechanism would solve those issues: every X amount of time, the producer
> produces a small no-op heartbeat message that tells the consumer it's still
> there; failure to receive one of those messages after N heartbeat intervals
> indicates that the producer is no longer alive.  The only thing it doesn't
> protect you against is temporary network perturbations (since they would
> make it appear that the producer is gone when it's not really), but you
> might be able to live with that.
>
> I'm not sure that heartbeats are a better solution than processing the
> advisory messages as you've suggested, but I think at a minimum you should
> have an EOF that's always sent (during normal operations) when the producer
> closes, and to which your consumer reacts by disconnecting as well.  Then
> the advisory messages can be used only for handling truly exceptional
> conditions.
>
> On Mon, Feb 2, 2015 at 11:29 AM, Kevin Burton <burton@spinn3r.com> wrote:
>
> > > ActiveMQ provides no means to guarantee that a producer will fail to
> > > deliver
> > > a message when there are no consumers interested in the message.  It is
> > > actually designed more for cases of consumers being away and returning
> > at a
> > > later time.
> > >
> > >
> > Perhaps. But I think the advisory queue message
> >
> > ActiveMQ.Advisory.NoConsumer.Queue
> >
> > … actually solves my problem.
> >
> > If a producer creates a queue, and no one is consuming, I’ll just create
> a
> > new consumer once I get the advisory message about the queue.
> >
> > This is actually how I do it *now* except I was listening to NEW queues…
> >
> > I’ve actually gotten a lot of use out of the advisory queue system.
> >
> >
> > > As you noted correctly, ActiveMQ automatically recreates destinations
> > every
> > > time they are needed, so even if they were GC'ed, they would just be
> > > automatically recreated when a producer posted a message.
> > >
> >
> > Yes.  And in my case, I think that’s ok.
> >
> > I might actually write up a blog post about this design. It’s novel but
> AMQ
> > actually does seem to be working out well for us without the limitations
> > you describe due to advisory messages.
> >
> > Kevin
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>



-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message