Hi Rob,

           Thanks for writing. Please see inline.

On Fri, Jan 20, 2012 at 1:27 AM, Rob Godfrey <rob.j.godfrey@gmail.com>wrote:

> Hi Praveen,
>
> apologies for the slow response here...
>
> On 14 January 2012 03:27, Praveen M <lefthandmagic@gmail.com> wrote:
>
> > Hi,
> >
> >    I would like to exploit the Berkeley DB's nosyncCommit (lazy commit)
> > option to exploit enqueue performance.
> > I see that in the current BDBMessageStore implementation there is no way
> to
> > override the syncCommit Boolean value and
> > would like to know if this could be made into an option.
> >
> > Also, that said It'll be nice to know if someone had used the
> > nosyncCommit() option that BDB provides and if using it does any
> invisible
> > harm. (DB corruptions etc.).
> >
> >
> So, I have coincidentally been working on a patch to increase throughput
> for non transactional messages which makes greater use of no-sync commits.
> The patch is basically to allow a certain amount of "out-of-order"
> processing while still maintaining the guarantees on durability that the
> AMQP specification states that you should get.  For non transactional
> durable messages this is giving me something like a 6.5x increase in max
> total throughput.
>

*Ah, this is great. Exactly what I was looking for. *

>
> Unfortunately this patch did uncover a couple of issues, one being the need
> to have some code in Qpid to handle occasional BDB lock timeouts... and the
> other a very occasional bug in BDB itself which they guys at Oracle now
> have a fix for, but won't be publicly released until their regular update.
> There is a reasonably easy workaround for this which I shall incorporate
> into my patch, but it may take me a couple more days to get it all
> complete.
>
> Note that my patch is a little different to what you are suggesting - in
> your case a message may go on to a queue in memory and then be delivered
> without the guarantee that it was sync'd to disk... in my patch the message
> won;t be released into the in-memory queue until it is sync'd, however I am
> batching up more such enqueues to get better performance.  In my tests the
> total max throughput with these two possible solutions is pretty much
> identical, however the latency on individual messages is likely to be a
> little different.
>

*That sounds even better. Your solution fits the bill perfectly.*
*I'd  prefer the message getting sync'd to disk first before delivery too.*
* *

>
> Now, to why do I want to override that value to false? (I do realize that
> > turning it to false, doesn't guarantee that the message was written to
> > disk)
> > In my use case. I have a database outside Qpid which persists all the
> > messages enqueued (for durability and transactionality with message
> > callback operations.).
> >
> > My sequence of operations is,
> >
> > 1) Enqueue messages -> persist message in my database outside Qpid ->
> > enqueue to Qpid in persistant mode
> > Since my message is already persisted in my database, I'm ok with losing
> a
> > message in BDB, and would like to enhance throughput of my overall
> enqueue
> > operation by using the nosyncCommit option.
> > The reason I'm not using non-persistant mode in Qpid is because I don't
> > want to overwhelm the memory available by flooding the broker with
> > messages.
> >
> > In the event of a broker crash, I plan to restore the messages from my
> > database outside Qpid. (I realize this could turn out a little
> expensive).
> > I've been thinking
> > that if there is a replication/HA solution to have messages go to a
> primary
> > and secondary broker (in a 2 instance cluster), in the event of a crash,
> > I'll have to
> > only restore the most recent messages from my database outside Qpid upto
> a
> > certain checkpoint to which the primary and secondary are sync'ed (Not
> sure
> > if this is achievable.).
> >
> > I do realize that my architecture is a little complicated. Please do let
> me
> > know if you have more questions about what I'm trying to achieve here,
> I'll
> > be happy to
> > clarify doubts.
> >
> > I'd really like to know if I can add an option to override the syncCommit
> > to false without any major side-effects apart from what I expect.
> >
> >
> >
> Does my patch seem like it would satisfy your requirements?
>

*Yes. It's quite exactly what I was looking for. I have a couple of
questions on the HA*
*state of things though. Please do see our other thread on that. It will be
really *
*valuable if you mention what you think.*
*
*
*Thank you Rob.*
* *

>
> Cheers,
> Rob
>



-- 
-Praveen