qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robbie Gemmell <robbie.gemm...@gmail.com>
Subject Re: JMS Client usage - best practices?
Date Wed, 11 Jan 2012 14:58:02 GMT
Hi Praveen,

If you are predominantly going to use persistent messages, then I
would actually look at your publishers using a single single session
per connection for the most part.

The broker is currently able to best extract performance scaling from
the persistent stores (more so the BDBstore as per Robs recent
testing...though the results of which are again slightly out of date
now due to the upgrade to BDB5 earlier today) by performing actions on
a per-connection basis, so it can extract parrallism over commits to
the store. Using a lower number of connections with 5-10 sessions each
could be noticably slower at peak when using 150-300 publishers.

If you are sending to a huge variety of Destinations then the way you
are using your Producers seems appropriate; creating producers isnt
necessarily that heavyweight an operation, but not doing so is
undeniably far more efficient.

I have actually never personally used sync_publish, but given that
both the underlying sync() call and session.commit() hit the broker
synchronously I would guess their performance should be pretty similar
for persistent messages with a batch size of 1, in which case I would
probably just go with transactions for 'pure JMS' purposes. If you
ever were using batches of >1 message that would seem to favor use of
transactions due to lowering the number of synchronous round trips to
the broker which would be required, increasing performance.

Robbie

On 10 January 2012 17:48, Praveen M <lefthandmagic@gmail.com> wrote:
> Hi Robbie,
>
> Thanks for writing.
>
> Here is some more context for my 2nd question.
>
> I'm expected to have about 30 client instances connecting to one broker.
> About 90% of the messages to the broker are expected to be persistent
> messages.
>
> I haven't decided whether to use transactions or use the option
> sync_publish. The
> guarantee that we'd expect is that the broker receives the message before
> proceeding to
> send the next message. I'd be happy to hear your recommendations around
> this.
>
> From the code I read, it looked like sync_publish was working to that
> effect. But one thing I
> saw was the sync() call in the client held to a connection failover lock,
> and I wasn't sure if the sync() call was blocking.
> Anyways, that said do you have a recommendation between transacted vs
> sync_publish in a multi client environment (things going in parallel). I'd
> really like to hear what you think about this.
>
> There will not be any batch enqueues (a top of my head). Almost all our
> traffic will be having messages
> enqueued one at a time.
>
> The number of destinations we have could be about 4000-5000. But yes, there
> is a strong likelihood of one
> queue being more popular and all the traffic getting routed there. We
> sometimes receive bursty enqueues of a certain type
> and is fair to assume that 5 or more enqueues of the same type to the same
> queue is possible to happen concurrently.
>
> We plan to have own framework on top of Qpid which will have the ability to
> load balance between the destinations
> if one queue goes over a certain threshold (some hand-picked number..not
> decided on yet) and then route traffic to another destination.
>
> And the overall traffic expected is somewhere between 10 million - 20
> million messages a day maybe.
>
> Please do let me know if you'd like to hear any other information.
>
> Thank you,
> Praveen
>
>
> But yes, we'd be using one of the two (whichever would perform best with
> multiple concurrent clients).
>
> On Tue, Jan 10, 2012 at 6:41 AM, Robbie Gemmell <robbie.gemmell@gmail.com>wrote:
>
>> Hi Praveen,
>>
>> Using either JNDI or Session.createQueue should work fine, but the
>> obvious limitation of the latter is that the (possibly
>> provider-specific) configuration is then part of your code (unless you
>> look the information up somewhere before using it) and cant be changed
>> without modifying the code.
>>
>> To better answer the other question, it would be good to first know
>> more specifics about the application and its messaging model, eg:
>> How many instances of the application are you likely to be running?
>> Will all your messages be persistent?
>> Will you always use transactions (and if so will it always of batch
>> size 1 like below)?
>> You mention having maybe 5-10 concurrent publishers, but how many
>> different Destinations are you likely to be sending to in total?
>> Are any of the above Destinations likely to be used a lot more than the
>> others?
>>
>> Robbie
>>
>> On 10 January 2012 00:27, Praveen M <lefthandmagic@gmail.com> wrote:
>> > Hi,
>> >
>> >   I'm writing a JMS Client and have a few best practices questions.
>> >
>> > 1) I understand that there are different ways ways to create a reference
>> to
>> > a destination to enqueue a message (using JMS createQueue, using JNDI
>> > lookup).
>> >    What would be the recommended way to use?
>> >
>> > 2) I create a connection to the broker on my application startup, and
>> > create a session pool. on every enqueue, i checkout a session from my
>> pool
>> > use it and return to pool on enqueue.
>> >    Does this sound reasonable?  (code sample below)
>> >
>> >    What would be the pattern that is typically used/recommended for
>> > session management/connection management in the client side?
>> >
>> >    Or is this an overkill to do? Is CreateSession() and CreateProducer()
>> > really cheap that it is just done on each enqueue?
>> >    In my use case, I'd expect quite some concurrency (assume n (maybe
>> > 5-10) threads doing enqueues at the same time) and I'd prefer it to not
>> > lock/block and go through asap.
>> >
>> >    I do remember from reading the qpid client code that there is a lock
>> > held on the connection object whenever a new session is created on that
>> > connection. I'm assuming it is one of the places where my enqueue can
>> slow
>> > down
>> >    when going in parallel if it has to create a session for every enqueue
>> > of a message. There might be more such blocks, which I'm not aware of.
>> >
>> >   Taking performance into mind, what would be the best design for
>> > session/connection management?
>> >
>> >    A snippet of what I have currently is below,
>> >
>> >    // a thread safe queue with the available session producer pairs which
>> > can be used to create a message and enqueue.
>> >    ConcurrentLinkedQueue<Pair<Session, MessageProducer>>
>> > availableSessionProducerPairs = new ConcurrentLinkedQueue<Pair<Session,
>> > MessageProducer>> ();
>> >
>> >   // initialize the sessionProducerPairs, create upto n pairs
>> >    initializePool(Connection qpidConnection) {
>> >         for (int i = 1 ; i <= SESSION_PRODUCER_PAIRS_COUNT; i ++ ) {
>> >            Session session = qpidConnection.createSession(true,
>> > Session.SESSION_TRANSACTED);
>> >            MessageProducer producer = session.createProducer(null);
>> >            availableSessionProducerPairs.add(Pair.newPair(session,
>> > producer));
>> >        }
>> >    }
>> >
>> > // on enqueue we checkout a session/producer pair use it for the enqueue
>> > and then return it back to the session/produer pair pool.
>> >   enqueueMessage(String queueName, byte[] message) {
>> >            // remove the session producer pair
>> >             Pair<Session, MessageProducer> sessionProducerPair =
>> > availableSessionProducerPairs.poll();
>> >        if(sessionProducerPair == null) {
>> >            // gack here and return null.
>> >        }
>> >        Session session = sessionProducerPair.getFirst();
>> >        MessageProducer producer = sessionProducerPair.getSecond();
>> >        BytesMessage jmsMessage = session.createBytesMessage();
>> >        jmsMessage.writeBytes(message);
>> >        Queue queueRef = queueNameToReferenceMap.get(queueName);
>> >        producer.send(queueRef, jmsMessage, DeliveryMode.PERSISTENT, 0,
>> 0);
>> >        session.commit();
>> >        availableSessionProducersPair.add()
>> >  }
>> >
>> >
>> >
>> > Thank you,
>> > -Praveen
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>
>
> --
> -Praveen

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message