activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: Marking a destination read-only?
Date Thu, 05 Feb 2015 02:35:38 GMT
On Wed, Feb 4, 2015 at 3:15 PM, artnaseef <art@artnaseef.com> wrote:

> Hey Kevin - I really think there's a better way to address your use case
> needs with ActiveMQ that would avoid the turnover of destinations.  I would
> welcome the opportunity to help you figure that out.
>
>
Thank.  I really appreciate it.

The only thing is I think it would be 2-4 pages of text explaining how it
works :-P

It’s indeed a rare use case.  We use ActiveMQ to power our ‘scheduler’
which is a traffic shaper for our web and social media crawler.

Essentially a URL is computed to a ‘throttle key’ which is one of the
randomly chosen IPs it resolves to.

Then we enqueue a message representing a task to fetch this content, into a
‘pending’ queue which is just

task_name:pending:ip

We then service this queue but we are a slow consumer because we use a
token bucket filter to traffic shape that IP so we don’t attempt to
overwhelm the site.

Overwhelming someone’s Internet resources and website is rude :-P

So if we get a token, which are handed out at 1 per second by default,
then, in a JMS transaction, we copy the message from the pending queue to
the serving queue.

All our robots just consume the serving queue.

This is essentially why we require lots of queues, some of which may go
away.

I’ve thought about other designs and the all mostly don’t work for one
reason or another.

For example, one strategy is a log, you stick tasks in the 0 queue, if you
don’t have capacity, you move them to the 1 queue, and so forth.  This
would give you less queues but you have no way to compute the depth of a
queue for a specific IP easily.

With that said, if you want to implement your own "read-only" destination
> feature, it would be relatively easy to do with a Broker Filter.
>
>
Cool.. if we end up developing one I’ll open source it and contribute it
back.


> One concern I have is putting too much optional functionality directly into
> ActiveMQ - every option increases the complexity of ActiveMQ and makes it
> that much harder to maintain in general.  "Separation of concerns" is a
> powerful tool.
>
>
I totally agree.  However, I think some of the primitives, like marking a
queue read only , should go in as they’re more of the unix and distributed
systems primitives.  For example, if you make a queue read only, it’s then
immutable and you can reason about it easier.

If anything I think ActiveMQ could do with some code pruning.

For example, it has JXTA support!  I think JXTA is officially dead isn’t
it? :-P

Maybe an exercise about what should be removed from ActiveMQ?

I mean I think this is PART of the goal of Apollo of course but it seems
that’s a ways away…

Kevin

-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message