qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen M <lefthandma...@gmail.com>
Subject Re: override to turn BDBMessageStore syncCommit to false?
Date Sat, 28 Jan 2012 01:42:23 GMT
Hi Rob,


Sorry for the late reply. I finally got around to reading your changelist
and also playing with it.
I realize that the change you've done is for sessions with auto_commit mode.

I was wondering if the async override could be applied to transacted mode
too.

If you read my earlier mail,

" My sequence of operations is,
 1) Enqueue messages -> persist message in my database outside Qpid ->
 enqueue to Qpid in persistant mode
 Since my message is already persisted in my database, I'm ok
with losing amessage in BDB, and would like to enhance throughput of my
overall enqueue"

Though persisting the message in my database, the reason why I'm using
transacted session is to protect against a case like below.

If I use auto_commit, the sequeue of operations are.
1) my enqueue server -> writes to my database outside qpid -> enqueue to
qpid (auto_commit) -> commit my database write outside qpid

But there is a case where, after i do the enqueue to qpid and commit my
database write,  my enqueue server can crash and the auto_committed message
might be lost, as there is
no guarantee that it has reached the broker (could be still in network
buffer or in transit..but not yet reached the broker).

To avoid this scenario I was thinking of two options.

To use publish_sync option with auto_commit
1) my enqueue server -> writes to my database outside qpid -> enqueue to
qpid (auto_commit with publish_sync) -> commit my database write outside
qpid

The downside to this is, publish_sync isn't really a JMS terminology and
also, on the client side, there is a lock held on the connection with
publish_sync option, avoiding other sessions sharing the connection to make
progress. I spoke to Robbie about this earlier, and his suggestion was to
try and stick to JMS terminology if possible.

2) Use transacted mode commit
 my enqueue server -> writes to my database outside qpid -> enqueue to qpid
(transacted_mode)  -> commit qpid session-> commit my database write
outside qpid

This looked pretty reliable and conforms to JMS, and I was hoping to use
this, as there is a guarantee that the message is pushed to the broker
before I write to my own database.
The one enhancement that I thought will improve the performance of the
roundtrip for this case was, to set syncCommit to false, so that I get the
guarantee that the broker got my message, but also return back quickly than
wait for the write to happen(which takes most of the roundtrip time in the
transaction).

I do understand that what I am trying to achieve is quite different from
normal usage.

I'd like to hear what you think about this problem and what you would
suggest as my best option in this case.

Thanks,
Praveen



Thanks,
Praveen


On Sat, Jan 21, 2012 at 12:00 PM, Rob Godfrey <rob.j.godfrey@gmail.com>wrote:

> Hi Praveen,
>
> I've checked in my changes on trunk if you want to have a play around...
> let me know if you run into any issues,
>
> Cheers,
> Rob
>
> On 20 January 2012 16:57, Praveen M <lefthandmagic@gmail.com> wrote:
>
> > Hi Rob,
> >
> >           Thanks for writing. Please see inline.
> >
> > On Fri, Jan 20, 2012 at 1:27 AM, Rob Godfrey <rob.j.godfrey@gmail.com
> > >wrote:
> >
> > > Hi Praveen,
> > >
> > > apologies for the slow response here...
> > >
> > > On 14 January 2012 03:27, Praveen M <lefthandmagic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > >    I would like to exploit the Berkeley DB's nosyncCommit (lazy
> commit)
> > > > option to exploit enqueue performance.
> > > > I see that in the current BDBMessageStore implementation there is no
> > way
> > > to
> > > > override the syncCommit Boolean value and
> > > > would like to know if this could be made into an option.
> > > >
> > > > Also, that said It'll be nice to know if someone had used the
> > > > nosyncCommit() option that BDB provides and if using it does any
> > > invisible
> > > > harm. (DB corruptions etc.).
> > > >
> > > >
> > > So, I have coincidentally been working on a patch to increase
> throughput
> > > for non transactional messages which makes greater use of no-sync
> > commits.
> > > The patch is basically to allow a certain amount of "out-of-order"
> > > processing while still maintaining the guarantees on durability that
> the
> > > AMQP specification states that you should get.  For non transactional
> > > durable messages this is giving me something like a 6.5x increase in
> max
> > > total throughput.
> > >
> >
> > *Ah, this is great. Exactly what I was looking for. *
> >
> > >
> > > Unfortunately this patch did uncover a couple of issues, one being the
> > need
> > > to have some code in Qpid to handle occasional BDB lock timeouts... and
> > the
> > > other a very occasional bug in BDB itself which they guys at Oracle now
> > > have a fix for, but won't be publicly released until their regular
> > update.
> > > There is a reasonably easy workaround for this which I shall
> incorporate
> > > into my patch, but it may take me a couple more days to get it all
> > > complete.
> > >
> > > Note that my patch is a little different to what you are suggesting -
> in
> > > your case a message may go on to a queue in memory and then be
> delivered
> > > without the guarantee that it was sync'd to disk... in my patch the
> > message
> > > won;t be released into the in-memory queue until it is sync'd, however
> I
> > am
> > > batching up more such enqueues to get better performance.  In my tests
> > the
> > > total max throughput with these two possible solutions is pretty much
> > > identical, however the latency on individual messages is likely to be a
> > > little different.
> > >
> >
> > *That sounds even better. Your solution fits the bill perfectly.*
> > *I'd  prefer the message getting sync'd to disk first before delivery
> too.*
> > * *
> >
> > >
> > > Now, to why do I want to override that value to false? (I do realize
> that
> > > > turning it to false, doesn't guarantee that the message was written
> to
> > > > disk)
> > > > In my use case. I have a database outside Qpid which persists all the
> > > > messages enqueued (for durability and transactionality with message
> > > > callback operations.).
> > > >
> > > > My sequence of operations is,
> > > >
> > > > 1) Enqueue messages -> persist message in my database outside Qpid
->
> > > > enqueue to Qpid in persistant mode
> > > > Since my message is already persisted in my database, I'm ok with
> > losing
> > > a
> > > > message in BDB, and would like to enhance throughput of my overall
> > > enqueue
> > > > operation by using the nosyncCommit option.
> > > > The reason I'm not using non-persistant mode in Qpid is because I
> don't
> > > > want to overwhelm the memory available by flooding the broker with
> > > > messages.
> > > >
> > > > In the event of a broker crash, I plan to restore the messages from
> my
> > > > database outside Qpid. (I realize this could turn out a little
> > > expensive).
> > > > I've been thinking
> > > > that if there is a replication/HA solution to have messages go to a
> > > primary
> > > > and secondary broker (in a 2 instance cluster), in the event of a
> > crash,
> > > > I'll have to
> > > > only restore the most recent messages from my database outside Qpid
> > upto
> > > a
> > > > certain checkpoint to which the primary and secondary are sync'ed
> (Not
> > > sure
> > > > if this is achievable.).
> > > >
> > > > I do realize that my architecture is a little complicated. Please do
> > let
> > > me
> > > > know if you have more questions about what I'm trying to achieve
> here,
> > > I'll
> > > > be happy to
> > > > clarify doubts.
> > > >
> > > > I'd really like to know if I can add an option to override the
> > syncCommit
> > > > to false without any major side-effects apart from what I expect.
> > > >
> > > >
> > > >
> > > Does my patch seem like it would satisfy your requirements?
> > >
> >
> > *Yes. It's quite exactly what I was looking for. I have a couple of
> > questions on the HA*
> > *state of things though. Please do see our other thread on that. It will
> be
> > really *
> > *valuable if you mention what you think.*
> > *
> > *
> > *Thank you Rob.*
> > * *
> >
> > >
> > > Cheers,
> > > Rob
> > >
> >
> >
> >
> > --
> > -Praveen
> >
>



-- 
-Praveen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message