qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robbie Gemmell <robbie.gemm...@gmail.com>
Subject Re: Batch/Bulk receive messages using java client?
Date Mon, 16 Jul 2012 11:40:28 GMT
Hi Praveen,

I have talked this over with some of the others here, and tend to agree
with Gordon and Rajith that mixing asynchronous and synchronous consumers
in that fashion isn't a route I would really suggest; using two sessions
makes for complication around transactionality and ordering, and I dont
think it will work on a single session.

We do have some ideas you could potentially use to implement batching in
the application to improve performance, but there are various subtleties to
consider that might heavily influence our suggestions. As such we really
need a good bit more detail around the use case to actually give a reasoned
answer. For example:

- How many connections/sessions/consumers/queues are actually in use?
- Are there multiple consumers on each/any of the queues at the same time?
- What if any ordering requirements are there on the message processing
(either within each queue or across all the queues)?
- What is the typical variation of message volumes across the queues that
you are looking to balance?
- What are the typical message sizes?
- How many messages might you potentially be looking to batch?
- What is the typical processing time in onMessage() now? Would this vary
as a direct multipe of the number of messages batched, or by some other
scaling?

Regards,
Robbie

On 12 July 2012 17:53, Praveen M <lefthandmagic@gmail.com> wrote:

> Hi,
>
> I'm trying to explore if there are ways to batch message processing.
> Batching message processing would help us improve performance for some of
> our use cases,
> where we could chunk messages and process them in a single callback.
>
> Have anyone here explored building a layer to batch messages.
>
> I am using the Java Broker and the Java client.
>
> I would like to stick to the JMS api as much as possible.
>
> This is what I currently have, still wondering if it'd work.
>
> 1) When the onMessage() callback is triggered, create a consumer a pull
> more messages to process from the queue where the message was delivered
> from.
> 2) Pull messages upto the number of my max chunk size, or upto the messages
> available in the queue.
> 3) process all the messages together and commit on the session.
>
> I'd like to hear ideas on how to go about this.
>
> Thanks,
> --
> -Praveen
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message