qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen M <lefthandma...@gmail.com>
Subject Re: JMS Client usage - best practices?
Date Fri, 13 Jan 2012 20:07:01 GMT
Thanks for writing Robbie. That was a lot of useful info.

I sure didn't think of the multiple consumer - single session case. Thanks
for the tip.

I'll try spreading by consumers across 2 connections and see if that helps.

Thank you,
Praveen

On Fri, Jan 13, 2012 at 10:10 AM, Robbie Gemmell
<robbie.gemmell@gmail.com>wrote:

> Hi Praveen,
>
> 180 connections should also be entirely fine, I have used several
> hundred. The only limit is going to be memory based.The key issue with
> using large amounts of connections currently is that when we moved
> away from Mina as the IO layer for 0.14 we moved to a model that uses
> 2 threads per connection, so there are a large number of Threads
> created which in turn will require memory for their stacks and heap
> for the objects they contain. Additionally, there are currently some
> large buffers placed in the heap on a per-connection basis that can
> quickly mount up. We are going to look at the impact those have on
> performance at particular buffer sizes and see if we can reduce the
> default, but it can certainly be reconfigured to meet a particular
> users requirements.
>
> With such a low throughput of <=1msg/s per consumer, there is vastly
> reduced scope for write-combining the store updates and so the use of
> a single connection becomes far less of issue. There will be scope to
> write-combine with other consumer instances and also the various
> producers, but it may still be worth looking at something inbetween
> the extremes of 20 Consumers per Connection and 1 Consumer per
> Connection (eg 10 Consumers per Connection), to see if it is possible
> to extract some scalability.
>
> With your slow onMessage handler the thing you should be most worried
> about is avoiding using multiple Consumers on the same Session,
> because each Session only has a single thread performing the
> deliveries. You will want to make sure you use a Session per Consumer.
>
> Robbie
>
> On 11 January 2012 19:02, Praveen M <lefthandmagic@gmail.com> wrote:
> > ah, my bad... I expect to have 20 consumer threads per instance. Sorry,
> so
> > that'd be 600 consumer threads in the environment.
> >
> > On Wed, Jan 11, 2012 at 10:14 AM, Praveen M <lefthandmagic@gmail.com>
> wrote:
> >
> >> Great, that helps.
> >>
> >> Would you have a similar recommendation for the consumer side?
> >>
> >> In my environment, I expect about 200 consumer threads per instance.
> Total
> >> (30 instance * 200 threads/per instance) - 6000 consumer threads
> >>
> >> I was thinking I'd just use one connection per instance  and then spin
> off
> >> 200 sessions/consumers.
> >>
> >> my onMessage() callback is expected to take atleast 1 second to process
> (I
> >> don't see a reason why a message should be submitted for async
> processing
> >> if it takes lesser than that to process)
> >> , so I assumed I must be ok with  one connection.
> >>
> >> A  latency is totally acceptable before a message gets picked up.
> >> Is there any recommendations towards connection management per
> >> instance for the consumer side of things? I'd like to hear what you
> think
> >> about it. If I'm right to assume 1 connection per
> >> instance should be good?
> >>
> >> Also,
> >> Since the consumer side would add more parallel connections to the
> broker,
> >> I'm curious upon the limits I could push the broker.
> >> Is there any tested limit in terms of the number of connections that the
> >> broker could support?
> >> We are looking at 180 concurrent connections to the broker if I say 1
> >> connection per instance for all the consumers and 5 connections per
> >> instance for enqueues Should that be ok?
> >>
> >> Also, are there any cons for having a lot of connections to the broker
> >> which I should be aware of?
> >>
> >> Thanks a lot,
> >> Praveen
> >>
> >>
> >> On Wed, Jan 11, 2012 at 9:52 AM, Robbie Gemmell <
> robbie.gemmell@gmail.com>wrote:
> >>
> >>> 150 connections should be no problem.
> >>>
> >>> I would expect any locking on the client to be of little significance
> >>> to overall performance when using persistent messages, hitting the
> >>> disk is going to dominate overall performance. The issue with using
> >>> multiple Sessions on a Connection with persistent messages is that it
> >>> the Sessions are ultimately multiplexed onto the same TCP connection
> >>> and their data processed sequentially on arrival at the broker. If for
> >>> example you were to do 5 concurrent sends on 5 Sessions on the same
> >>> Connection, they will be processed onto the connection sequentially
> >>> and thus into the store sequentially upon arrival at the broker,
> >>> whereas if you do 5 concurrent sends for Sessions on individual
> >>> Connections there is scope for them to be processed into the store
> >>> concurrently and enable some/all of them to be write-combined and
> >>> synced to disk at the same time, reducing the required synchronous
> >>> disk IO to perform the operations and thus providing increased overall
> >>> performance/scalability by effectively 'doing more with less' so to
> >>> speak.
> >>>
> >>> Robbie
> >>>
> >>> On 11 January 2012 16:24, Praveen M <lefthandmagic@gmail.com> wrote:
> >>> > Hi Robbie,
> >>> >
> >>> > Thanks for writing. I just had a few followup questions. Please see
> >>> in-line.
> >>> >
> >>> > Thanks a lot for helping out with this,
> >>> > Praveen
> >>> >
> >>> > On Wed, Jan 11, 2012 at 6:58 AM, Robbie Gemmell <
> >>> robbie.gemmell@gmail.com>wrote:
> >>> >
> >>> >> Hi Praveen,
> >>> >>
> >>> >> If you are predominantly going to use persistent messages, then
I
> >>> >> would actually look at your publishers using a single single session
> >>> >> per connection for the most part.
> >>> >>
> >>> >> The broker is currently able to best extract performance scaling
> from
> >>> >> the persistent stores (more so the BDBstore as per Robs recent
> >>> >> testing...though the results of which are again slightly out of
date
> >>> >> now due to the upgrade to BDB5 earlier today) by performing actions
> on
> >>> >> a per-connection basis, so it can extract parrallism over commits
to
> >>> >> the store. Using a lower number of connections with 5-10 sessions
> each
> >>> >> could be noticably slower at peak when using 150-300 publishers.
> >>> >>
> >>> >
> >>> > that totally makes sense. But now that I'd have approximately (30
> >>> instance
> >>> > * 5 connections) = 150
> >>> > connections to the broker, I wouldn't be overwhelming the broker
> right?
> >>> I'm
> >>> > guessing it shouldn't be a problem, But just want to run it by you?
> >>> >
> >>> > I understand that having separate connections is significantly
> faster as
> >>> > each
> >>> > connection would operate on it's own socket.
> >>> > Just out of curiosity I'd like to know, what are the performance
> >>> > degradation
> >>> > causes when using sessions sharing the same connection? I know for
> one
> >>> that
> >>> > the session has to hold on creation locks (for creation) and
> connection
> >>> > failover locks on send.
> >>> > Do you think this will be the cause of the major bottle necks or is
> >>> there
> >>> > something else too?
> >>> > (I'm sure you know the code back to front to answer this better) :)
> >>> >
> >>> >
> >>> >
> >>> >> If you are sending to a huge variety of Destinations then the way
> you
> >>> >> are using your Producers seems appropriate; creating producers
isnt
> >>> >> necessarily that heavyweight an operation, but not doing so is
> >>> >> undeniably far more efficient.
> >>> >>
> >>> >
> >>> > ah, yep. that makes sense. I think I will stick to creating
> producers on
> >>> > connection pooling
> >>> > and creating the session
> >>> >
> >>> >
> >>> >>
> >>> >> I have actually never personally used sync_publish, but given that
> >>> >> both the underlying sync() call and session.commit() hit the broker
> >>> >> synchronously I would guess their performance should be pretty
> similar
> >>> >> for persistent messages with a batch size of 1, in which case I
> would
> >>> >> probably just go with transactions for 'pure JMS' purposes. If
you
> >>> >> ever were using batches of >1 message that would seem to favor
use
> of
> >>> >> transactions due to lowering the number of synchronous round trips
> to
> >>> >> the broker which would be required, increasing performance.
> >>> >>
> >>> >> yep, I totally agree with you on using transactions over the
> >>> sync_publish
> >>> > option
> >>> > for pure JMS purposes.
> >>> >
> >>> >
> >>> >> Robbie
> >>> >>
> >>> >> On 10 January 2012 17:48, Praveen M <lefthandmagic@gmail.com>
> wrote:
> >>> >> > Hi Robbie,
> >>> >> >
> >>> >> > Thanks for writing.
> >>> >> >
> >>> >> > Here is some more context for my 2nd question.
> >>> >> >
> >>> >> > I'm expected to have about 30 client instances connecting
to one
> >>> broker.
> >>> >> > About 90% of the messages to the broker are expected to be
> persistent
> >>> >> > messages.
> >>> >> >
> >>> >> > I haven't decided whether to use transactions or use the option
> >>> >> > sync_publish. The
> >>> >> > guarantee that we'd expect is that the broker receives the
message
> >>> before
> >>> >> > proceeding to
> >>> >> > send the next message. I'd be happy to hear your recommendations
> >>> around
> >>> >> > this.
> >>> >> >
> >>> >> > From the code I read, it looked like sync_publish was working
to
> that
> >>> >> > effect. But one thing I
> >>> >> > saw was the sync() call in the client held to a connection
> failover
> >>> lock,
> >>> >> > and I wasn't sure if the sync() call was blocking.
> >>> >> > Anyways, that said do you have a recommendation between
> transacted vs
> >>> >> > sync_publish in a multi client environment (things going in
> >>> parallel).
> >>> >> I'd
> >>> >> > really like to hear what you think about this.
> >>> >> >
> >>> >> > There will not be any batch enqueues (a top of my head). Almost
> all
> >>> our
> >>> >> > traffic will be having messages
> >>> >> > enqueued one at a time.
> >>> >> >
> >>> >> > The number of destinations we have could be about 4000-5000.
But
> yes,
> >>> >> there
> >>> >> > is a strong likelihood of one
> >>> >> > queue being more popular and all the traffic getting routed
> there. We
> >>> >> > sometimes receive bursty enqueues of a certain type
> >>> >> > and is fair to assume that 5 or more enqueues of the same
type to
> the
> >>> >> same
> >>> >> > queue is possible to happen concurrently.
> >>> >> >
> >>> >> > We plan to have own framework on top of Qpid which will have
the
> >>> ability
> >>> >> to
> >>> >> > load balance between the destinations
> >>> >> > if one queue goes over a certain threshold (some hand-picked
> >>> number..not
> >>> >> > decided on yet) and then route traffic to another destination.
> >>> >> >
> >>> >> > And the overall traffic expected is somewhere between 10 million
> - 20
> >>> >> > million messages a day maybe.
> >>> >> >
> >>> >> > Please do let me know if you'd like to hear any other information.
> >>> >> >
> >>> >> > Thank you,
> >>> >> > Praveen
> >>> >> >
> >>> >> >
> >>> >> > But yes, we'd be using one of the two (whichever would perform
> best
> >>> with
> >>> >> > multiple concurrent clients).
> >>> >> >
> >>> >> > On Tue, Jan 10, 2012 at 6:41 AM, Robbie Gemmell <
> >>> >> robbie.gemmell@gmail.com>wrote:
> >>> >> >
> >>> >> >> Hi Praveen,
> >>> >> >>
> >>> >> >> Using either JNDI or Session.createQueue should work fine,
but
> the
> >>> >> >> obvious limitation of the latter is that the (possibly
> >>> >> >> provider-specific) configuration is then part of your
code
> (unless
> >>> you
> >>> >> >> look the information up somewhere before using it) and
cant be
> >>> changed
> >>> >> >> without modifying the code.
> >>> >> >>
> >>> >> >> To better answer the other question, it would be good
to first
> know
> >>> >> >> more specifics about the application and its messaging
model, eg:
> >>> >> >> How many instances of the application are you likely to
be
> running?
> >>> >> >> Will all your messages be persistent?
> >>> >> >> Will you always use transactions (and if so will it always
of
> batch
> >>> >> >> size 1 like below)?
> >>> >> >> You mention having maybe 5-10 concurrent publishers, but
how many
> >>> >> >> different Destinations are you likely to be sending to
in total?
> >>> >> >> Are any of the above Destinations likely to be used a
lot more
> than
> >>> the
> >>> >> >> others?
> >>> >> >>
> >>> >> >> Robbie
> >>> >> >>
> >>> >> >> On 10 January 2012 00:27, Praveen M <lefthandmagic@gmail.com>
> >>> wrote:
> >>> >> >> > Hi,
> >>> >> >> >
> >>> >> >> >   I'm writing a JMS Client and have a few best practices
> >>> questions.
> >>> >> >> >
> >>> >> >> > 1) I understand that there are different ways ways
to create a
> >>> >> reference
> >>> >> >> to
> >>> >> >> > a destination to enqueue a message (using JMS createQueue,
> using
> >>> JNDI
> >>> >> >> > lookup).
> >>> >> >> >    What would be the recommended way to use?
> >>> >> >> >
> >>> >> >> > 2) I create a connection to the broker on my application
> startup,
> >>> and
> >>> >> >> > create a session pool. on every enqueue, i checkout
a session
> >>> from my
> >>> >> >> pool
> >>> >> >> > use it and return to pool on enqueue.
> >>> >> >> >    Does this sound reasonable?  (code sample below)
> >>> >> >> >
> >>> >> >> >    What would be the pattern that is typically used/recommended
> >>> for
> >>> >> >> > session management/connection management in the client
side?
> >>> >> >> >
> >>> >> >> >    Or is this an overkill to do? Is CreateSession()
and
> >>> >> CreateProducer()
> >>> >> >> > really cheap that it is just done on each enqueue?
> >>> >> >> >    In my use case, I'd expect quite some concurrency
(assume n
> >>> (maybe
> >>> >> >> > 5-10) threads doing enqueues at the same time) and
I'd prefer
> it
> >>> to
> >>> >> not
> >>> >> >> > lock/block and go through asap.
> >>> >> >> >
> >>> >> >> >    I do remember from reading the qpid client code
that there
> is a
> >>> >> lock
> >>> >> >> > held on the connection object whenever a new session
is
> created on
> >>> >> that
> >>> >> >> > connection. I'm assuming it is one of the places
where my
> enqueue
> >>> can
> >>> >> >> slow
> >>> >> >> > down
> >>> >> >> >    when going in parallel if it has to create a session
for
> every
> >>> >> enqueue
> >>> >> >> > of a message. There might be more such blocks, which
I'm not
> >>> aware of.
> >>> >> >> >
> >>> >> >> >   Taking performance into mind, what would be the
best design
> for
> >>> >> >> > session/connection management?
> >>> >> >> >
> >>> >> >> >    A snippet of what I have currently is below,
> >>> >> >> >
> >>> >> >> >    // a thread safe queue with the available session
producer
> >>> pairs
> >>> >> which
> >>> >> >> > can be used to create a message and enqueue.
> >>> >> >> >    ConcurrentLinkedQueue<Pair<Session, MessageProducer>>
> >>> >> >> > availableSessionProducerPairs = new
> >>> >> ConcurrentLinkedQueue<Pair<Session,
> >>> >> >> > MessageProducer>> ();
> >>> >> >> >
> >>> >> >> >   // initialize the sessionProducerPairs, create
upto n pairs
> >>> >> >> >    initializePool(Connection qpidConnection) {
> >>> >> >> >         for (int i = 1 ; i <= SESSION_PRODUCER_PAIRS_COUNT;
i
> ++
> >>> ) {
> >>> >> >> >            Session session = qpidConnection.createSession(true,
> >>> >> >> > Session.SESSION_TRANSACTED);
> >>> >> >> >            MessageProducer producer =
> >>> session.createProducer(null);
> >>> >> >> >
>  availableSessionProducerPairs.add(Pair.newPair(session,
> >>> >> >> > producer));
> >>> >> >> >        }
> >>> >> >> >    }
> >>> >> >> >
> >>> >> >> > // on enqueue we checkout a session/producer pair
use it for
> the
> >>> >> enqueue
> >>> >> >> > and then return it back to the session/produer pair
pool.
> >>> >> >> >   enqueueMessage(String queueName, byte[] message)
{
> >>> >> >> >            // remove the session producer pair
> >>> >> >> >             Pair<Session, MessageProducer>
sessionProducerPair
> =
> >>> >> >> > availableSessionProducerPairs.poll();
> >>> >> >> >        if(sessionProducerPair == null) {
> >>> >> >> >            // gack here and return null.
> >>> >> >> >        }
> >>> >> >> >        Session session = sessionProducerPair.getFirst();
> >>> >> >> >        MessageProducer producer =
> sessionProducerPair.getSecond();
> >>> >> >> >        BytesMessage jmsMessage = session.createBytesMessage();
> >>> >> >> >        jmsMessage.writeBytes(message);
> >>> >> >> >        Queue queueRef = queueNameToReferenceMap.get(queueName);
> >>> >> >> >        producer.send(queueRef, jmsMessage,
> >>> DeliveryMode.PERSISTENT, 0,
> >>> >> >> 0);
> >>> >> >> >        session.commit();
> >>> >> >> >        availableSessionProducersPair.add()
> >>> >> >> >  }
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > Thank you,
> >>> >> >> > -Praveen
> >>> >> >>
> >>> >> >>
> >>> ---------------------------------------------------------------------
> >>> >> >> Apache Qpid - AMQP Messaging Implementation
> >>> >> >> Project:      http://qpid.apache.org
> >>> >> >> Use/Interact: mailto:users-subscribe@qpid.apache.org
> >>> >> >>
> >>> >> >>
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > -Praveen
> >>> >>
> >>> >>
> ---------------------------------------------------------------------
> >>> >> Apache Qpid - AMQP Messaging Implementation
> >>> >> Project:      http://qpid.apache.org
> >>> >> Use/Interact: mailto:users-subscribe@qpid.apache.org
> >>> >>
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > -Praveen
> >>>
> >>> ---------------------------------------------------------------------
> >>> Apache Qpid - AMQP Messaging Implementation
> >>> Project:      http://qpid.apache.org
> >>> Use/Interact: mailto:users-subscribe@qpid.apache.org
> >>>
> >>>
> >>
> >>
> >> --
> >> -Praveen
> >>
> >
> >
> >
> > --
> > -Praveen
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>


-- 
-Praveen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message