activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Lots of small ActiveMQ instances or one big one?
Date Sat, 14 Mar 2015 14:32:15 GMT
I think the difference is that an ack packet (telling the sender that it
can discard the message because the broker now has it stored and the
sender's copy is no longer needed as a backup) is sent after a successful
write to disk, so the consequences of a failed disk write are different
than a failed Ethernet or queue write.

So if you batch disk writes, you either batch acks (which has a negative
performance impact; I'm not sure if the loss will be greater or less than
the gain from batching writes, but at a minimum it'll decrease the expected
improvement from batching writes) or you optimistically ack before the bits
hit the disk (which violates the JMS contract because it allows message
loss if the server crashes before the batch is written).
On Mar 13, 2015 7:57 PM, "Kevin Burton" <burton@spinn3r.com> wrote:

> > Regarding batching in my opinion it is right, even if is not the most
> performing solution, to have a fsync per msg as otherwise in case of outage
> you lose more messages
>
> To clarify, you get the same mathematical / functional properties of
> messages, just in the case of smart batching, you get MUCH better
> throughput.
>
> While the packet is over the ethernet it’s not sync’d .. while it’s waiting
> in queue, it’s not sync’d.
>
> The change is that instead of taking one message at a time, and writing,
> then syncing each one, you either read N messages at once, or the entire
> queue.
>
> The performance improvement can be dramatic. (100x).
>
>
> On Fri, Mar 13, 2015 at 5:59 PM, Christian Grassi <
> christiangrassi@gmail.com
> > wrote:
>
> > Hi Kevin,
> > just curios how many messages per second you are seeing?
> > Regarding batching in my opinion it is right, even if is not the most
> > performing solution, to have a fsync per msg as otherwise in case of
> outage
> > you lose more messages. Personally speaking in a persistent queue if the
> > producer is acknowledged that a message is in the queue it should be sent
> > and persisted.
> > Just my opinion
> >
> > Chris
> >
> > Il giorno ven 13 mar 2015 23:06 Kevin Burton <burton@spinn3r.com> ha
> > scritto:
> >
> > > Does ActiveMQ do an fsync per message?
> > >
> > > Why not do smart batching:
> > >
> > > http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
> > >
> > > This way the messages could be elided, and one fsync called on all of
> > them.
> > >
> > > Maybe it already does, but the comment that each messages requires a
> sync
> > > seems to hint that this could be more efficient.
> > >
> > > On Fri, Mar 13, 2015 at 2:46 PM, artnaseef <art@artnaseef.com> wrote:
> > >
> > > > I think they are in-line.  More producers generally mean more
> > concurrent
> > > > work
> > > > for the broker, as a single producer sends all of its messages
> serially
> > > to
> > > > the broker.  Similar logic applies to consumers.
> > > >
> > > > More destinations generally forces more producers and consumers,
> > although
> > > > it
> > > > doesn't have to.
> > > >
> > > > Is there something more specific that doesn't seem to match?
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > > http://activemq.2283324.n4.nabble.com/Lots-of-small-
> > > ActiveMQ-instances-or-one-big-one-tp4693133p4693203.html
> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Founder/CEO Spinn3r.com
> > > Location: *San Francisco, CA*
> > > blog: http://burtonator.wordpress.com
> > > … or check out my Google+ profile
> > > <https://plus.google.com/102718274791889610666/posts>
> > > <http://spinn3r.com>
> > >
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message