qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajith Muditha Attapattu <rajit...@gmail.com>
Subject Re: how to scale (out) qpid
Date Thu, 21 May 2015 15:33:54 GMT
Having used Dispatch with some POCs, I'd agree with Jack that it's at least
good to investigate that route ;).
If you want to know more about how you could handle the 0-10 routing model
while using dispatch I could provide some suggestions there.

IIRC there's also an effort to support this natively in dispatch. But I'm
not sure about the status of that yet.

Regards,

Rajith

On Wed, May 20, 2015 at 4:52 PM, Gibson, Jack <jagibson@paypal.com.invalid>
wrote:

> Olivier -  look at qpid dispatch router that¹s how I'd handle it.
>
> Jack
>
> Jack Gibson
> Chief Architect
> Core Payments Platforms @ PayPal
>
>
>
>
>
>
> On 5/20/15, 5:31 AM, "Olivier Mallassi" <olivier.mallassi@gmail.com>
> wrote:
>
> >Thank you rob. I will use non persitent messages for this.
> >
> >I like your idea of load-balancing at "message layer" instead of
> >"connection layer"! I need to figure out how to implement this, but this
> >is
> >clearly a more elegant solution (knowing we do not have thousands of
> >processes and connections)
> >
> >PS : for the record, we will also "functionally partitioned" : event of
> >type A goes to broker A (eg. a dedicated JVM), event of type B goes to
> >broker B
> >
> >On Wed, May 20, 2015 at 12:48 PM, Rob Godfrey <rob.j.godfrey@gmail.com>
> >wrote:
> >
> >> On 20 May 2015 at 12:00, Olivier Mallassi <olivier.mallassi@gmail.com>
> >> wrote:
> >> > Hi all
> >> >
> >> > my apologies in advance if I ask dummy questions and for this (maybe
> >>too)
> >> > long email but *i need some help in regards with "qpid clustering"*.
> >> > I have read some of the already asked questions on the ML and would
> >>like
> >> to
> >> > confirm (or not) some points.
> >> >
> >> > *Overview*:
> >> > I wanna use the Java Broker (mainly because I need to support several
> >>OS
> >> -
> >> > linux, solaris, windows....) but I am wondering our to scale out (or
> >>how
> >> to
> >> > cluster) qpid (target throughput will be around 150k events per sec,
> >> > message size variable but less than 1kB, connections number is not
> >>known
> >> > but dozens.
> >> > I saw some numbers on the ML (like 70k events per sec for a single
> >>broker
> >> > etc...) so it appears I will have to distribute my load on 2 or 3
> >> brokers.
> >> >
> >>
> >> Are you using persistent or non persistent messages?  The throughput
> >> the broker can achieve will obviously be very much tied to the type of
> >> messages and the sort of hardware you are using.
> >>
> >> > for now, I am not talking about HA.
> >> >
> >> > *1/ *I can cluster qipd using LVS or corosync. not ideal in my case
> >> because
> >> > it relies on OS specific solution
> >> >
> >> > *2/* I noticed that the messaging api uses the first IP in the
> >>specified
> >> > broker list (amqp://guest:guest@
> >> > /default?failover='roundrobin?cyclecount='2''&brokerlist=")
> >> >
> >> > So I can use a custom strategy (round robin, zookeeper based), coded
> >>in
> >> > java, to shuffle the brokers list and ensure to randomly distribute
> >>the
> >> > connections (when they are established). it works for fail over, can
> >>be
> >> > tricky if you want to be "elastic" but it should work.
> >> >
> >> > what are your thought on that approach?
> >>
> >> Depending on exactly how your application works, that sounds
> >> reasonable... you'd obviously need to ensure that every destination
> >> produced to had consumers on...
> >>
> >> >
> >> > *3/* I am yet facing an issue because in some cases, I need to use
> >>node
> >> > affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I
> >> need
> >> > to ensure the same consumer jvm will get it.
> >> > so I need to ensure that for a given JMSXgroupID, publisher & consumer
> >> use
> >> > the same broker (which begin to be tricky, even using hashcode, in
> >>case
> >> of
> >> > failover etc...)
> >>
> >>
> >> At this point it might seem like it would make more sense for every
> >> producer/consumer to have connections to every broker... and to do the
> >> round-robining at a higher lever than establishing the connection.
> >> The client can then make the choice on a per-message basis as to which
> >> broker to send to - I have seen this pattern used elsewhere.
> >>
> >> > Does federation (Dispatch Router) helps in that case (knowing it will
> >>add
> >> > network hops, and so latency)? Is there any getting started/tutorial
> >>that
> >> > can help better understanding the fixedAdress and linkRoutePattern?
> >> >
> >> > 4/ what about HA? It appears I will finish with a topology like
> >> >             |                | A.Master --> A.Replica
> >> > Client   |   "hash"  | B.Master --> B.Replica
> >> >             |                | C.Master --> C.Replica
> >> >
> >>
> >> Unfortunately I'm not very well acquainted with Dispatch - so I'll let
> >> others step in here.
> >>
> >> -- Rob
> >>
> >> >
> >> > Hope I was clear enough. many thanks for your help/feedbacks.
> >> > Regards
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> For additional commands, e-mail: users-help@qpid.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message