Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of gaurav.cs.sharma@gmail.com
 designates 209.85.220.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAZU44mCqawV+LgVWZp33ovr89MaEGzfcDtda6hL5V12WZmeFQ@mail.gmail.com>
References: 
 <CAAZU44mCqawV+LgVWZp33ovr89MaEGzfcDtda6hL5V12WZmeFQ@mail.gmail.com>
From: Gaurav Sharma <gaurav.cs.sharma@gmail.com>
Date: Thu, 29 May 2014 12:21:03 -0700
Message-ID: 
 <CAA=go7=CmJ5WbXnfJJLaXhq3Y2MMnMszQfc4tFRL4Z+1UX5XWA@mail.gmail.com>
Subject: Re: 1M queues?
To: "users@activemq.apache.org" <users@activemq.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Without knowing much about the domain, now that doesn't seem like the
best way to attack this problem. If you were hoping to do any sort of
secondly or few-secondly moving windows, the rpc/remoteness alone
would negate the use of such a system. A remote system for rate
throttling is better suited for more granular time windows. Also, a
busy system running on a decent machine with 32 or 48 cores, even with
H/T on, juggling 1M or even 100K persistent queues on a single
machine, is not going to work very cleanly and you might end up not
hitting even the internal sla's. If you absolutely must, partition it
away and then replicate the partitions.

In the system being throttled, can you assume some kind of fair load
balancing across machines receiving the to-be-throttled requests? If
yes, then a much more scalable option is to keep an in-memory rate
throttler implementation on all of the nodes with periodic
snapshotting of data points to disk/db if absolutely needed. You can
implement one easily using a deque or let me know and I can share my
implementation.


On May 29, 2014, at 11:19, Kevin Burton <burton@spinn3r.com> wrote:

How would activemq handle a lot of queues per box?  Like 1M =E2=80=A6

I'm doing this to implement throttling.  and since we can get stats on each
queue, I can see if we're massively backlogged on one of the queues.

AIs each queue its own database?

--=20

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
blog: http://burtonator.wordpress.com
=E2=80=A6 or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
War is peace. Freedom is slavery. Ignorance is strength. Corporations are
people.