activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: 1M queues?
Date Thu, 29 May 2014 19:51:11 GMT
I imagine I could keep everything in ONE queue… which makes things easier.
 It's use that at certain times, some items could be throttled into the
millions of items.


On Thu, May 29, 2014 at 12:21 PM, Gaurav Sharma <gaurav.cs.sharma@gmail.com>
wrote:

> Without knowing much about the domain, now that doesn't seem like the
> best way to attack this problem. If you were hoping to do any sort of
> secondly or few-secondly moving windows, the rpc/remoteness alone
> would negate the use of such a system. A remote system for rate
> throttling is better suited for more granular time windows. Also, a
> busy system running on a decent machine with 32 or 48 cores, even with
> H/T on, juggling 1M or even 100K persistent queues on a single
> machine, is not going to work very cleanly and you might end up not
> hitting even the internal sla's. If you absolutely must, partition it
> away and then replicate the partitions.
>
> In the system being throttled, can you assume some kind of fair load
> balancing across machines receiving the to-be-throttled requests? If
> yes, then a much more scalable option is to keep an in-memory rate
> throttler implementation on all of the nodes with periodic
> snapshotting of data points to disk/db if absolutely needed. You can
> implement one easily using a deque or let me know and I can share my
> implementation.
>
>
> On May 29, 2014, at 11:19, Kevin Burton <burton@spinn3r.com> wrote:
>
> How would activemq handle a lot of queues per box?  Like 1M …
>
> I'm doing this to implement throttling.  and since we can get stats on each
> queue, I can see if we're massively backlogged on one of the queues.
>
> AIs each queue its own database?
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> Skype: *burtonator*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
> War is peace. Freedom is slavery. Ignorance is strength. Corporations are
> people.
>



-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
War is peace. Freedom is slavery. Ignorance is strength. Corporations are
people.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message