activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas R. Corbin" <thomas.cor...@gmail.com>
Subject Re: Scalability of 'Networks of Brokers'
Date Tue, 09 Sep 2008 18:44:53 GMT
On Friday 05 September 2008, Bryan Murphy said:
> Sounds like a nightmare to me.  Maybe I don't understand what you're trying
> to accomplish, but I think something like Hadoop would be a better fit for
> this kind of problem.

sounds like a job that JINI was designed for.

>
> Bryan
>
> On Fri, Sep 5, 2008 at 12:10 PM, Martin_Cornelius <
>
> Martin.Cornelius@smiths-heimann.com> wrote:
> > Hi all,
> >
> > i am currently evaluating the scalability of Networks of ActiveMQ Brokers
> > with a large number of brokers. The actual drive behind this evaluation:
> > I have to design a distributed system that is completely serverless. It
> > shall be possible to switch of any of the machines in the system without
> > alloying the distributed functionality. The number of active machines in
> > the system might vary from only 2 to about 100.
> >
> > My current idea: Have an (presumably embedded) ActiveMQ Broker running on
> > any of the machines, and let all this Brokers form a 'Network of Brokers'
> > (abbreviated NOB in the follwing)
> >
> > So far, i found that in an NOB a message is forwarded only to those
> > brokers where at least one client has subscribed to the topic resp. queue
> > the message is sent to. Really fine. If a client subscribes to a queue,
> > the broker where the subscription is made sends a single short
> > control-message to all other brokers of the NOB. That would still allow
> > for thousands of brokers, fine again.
> >
> > However, i just realized one fact that may compromise scalabilty: If a
> > client subscribes to a  topic, the broker where the subscription is done
> > sends a not-so-short control-message to all other brokers. So far not too
> > bad, but now comes the problem: Every broker seems to react on this
> > control-message by sending another not-so-short control-message to all
> > other
> > brokers. This means, if N brokers are present, every subscription to a
> > topic
> > results in about (N-1)*N*2 control-messages (The additional factor two
> > comes
> > from the fact, that the final recipient-broker seems to send another
> > response control-message). BTW, The length of the control-messages being
> > sent seems to be closely related to the string length of the topic name.
> >
> > I have verified this behaviour with only up to 6 brokers, so i'm not
> > totally
> > shure if it will extrapolate to a larger number, but i suspect it does.
> > This
> > would mean, that with e.g. 100 Brokers every subscription to a topic
> > would result in 100*99*2 control messages being sent. If a single control
> > message has about 400 bytes (what i observed), this means a single
> > subscription to a
> > topic would result in about 8 Megabyte network traffic. This might even
> > be sustainable, but the quadratic order kills scalabilty -- with 1000
> > Brokers we would have to transfer about 1 Gigabyte, what is no longer
> > feasible.
> >
> > Finally, my questions: Is there any configuration option by which this
> > quadratic order can be avoided, is my idea simply foolish, or are my
> > conclusions wrong ? Note that i gained my conclusions more or less
> > 'empirically' using wireshark.
> >
> > thanks for any advice, Martin
> >
> > --
> > View this message in context:
> > http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699
> >p19335699.html Sent from the ActiveMQ - User mailing list archive at
> > Nabble.com.


-- 
Card #1. Uncle Bob’s Three Laws

• Write no production code except to pass a failing test.
• Write only enough of a test to demonstrate a failure.
• Write only enough production code to pass the test.


Mime
View raw message