qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Godfrey <rob.j.godf...@gmail.com>
Subject Re: qpid.periodic exchange type
Date Tue, 10 Nov 2009 03:33:11 GMT
2009/11/10 Aidan Skinner <aidan.skinner@gmail.com>

> On Sun, Nov 8, 2009 at 5:15 PM, Robert Godfrey <rob.j.godfrey@gmail.com>
> wrote:
> > 2009/11/8 Aidan Skinner <aidan@apache.org>
> >
> >> The original message is delivered and persisted as normal. The
> >> exchange would hold onto a copy of that message, and generate an
> >> entirely new, identical one at the specified interval.
> >>
> >>  Putting this logic at the exchange level seems better to me because
> >> it avoids further complicating the subscription implementation. It'll
> >> also map a bit cleaner into 1.0 semantics (as a non-destructive link).
> >>
> >>
> > I don't think we would need to complicate subscription implementation
> unless
> > we wanted to implement this by having selectors which compared message
> > fields to the present time (or some other condition that would need to be
> > periodically reevaluated).  I was thinking that we would implement like
> TTL
> > - i.e. the message state on the queue is changed when the time-driven
> > condition changes (in this case when it becomes live)...  As happens
> already
> > in the queue, the state change will cause the subscribers to go
> re-examine
> > the message to see if they can now deliver it.  Other than sorting out
> how
> > we do timed events a bit, this seems like a fairly minor change at the
> queue
> > level.
>
> I hadn't realised we could signal state change to the subscriber to
> have it reevaluate all the messages, that does make it much simpler.
> AIUI messages which had an expired TTL were cleaned up by the
> housekeeping thread after the pointer had already moved on.
>
>
Yes - this is basically how release works in general in the Java Broker
since other subscribers may have already advanced to a point beyond the
place where the released message exists.


> > Having exchanges that retain state would be a big departure, and as
> Robert
> > points out would complicate persistence.  From a 1-0 point of view, there
> > will be no such thing as exchanges, so I'm not sure I really see the
> > advantage there.  Basically the difference is between hard-coding the
> entry
> > point/address for the message (by fixing the exchange type) or adding a
> new
> Re-reading my mail I wasn't entirely clear about the use-case: this
> was someone who basically wanted to publish the same message every N
> minutes. Think cron with a really basic client that sends a message in
> a static file. the deliver-after stuff was a top level stab at doing
> it in AMQP.
>

OK, that is somewhat different, although from a 1-0 perspective you don't
really want to make them the "same" message as we say that queues should
reject duplicates.

>
> > capability to queues.  I much prefer the latter, and I also think it will
> be
> > cheaper to implement
>
> So, the main functional difference I see with an "X-deliver-after"
> header is that the publisher has to occasionally push a fresh batch of
> messages. Eventually all the messages would be consumed and they'd
> have to send a new batch. I'm not sure that I see much benefit in
> having a cron job that fires every 3 hours instead of every 15
> minutes.
>
> With the exchange implementation, they'd only need to publish 1. Well,
> 2. Somewhere in the back of my mind, with the exchange idea, I thought
> they'd publish a cancel message to stop the flow. That has a couple of
> problems from a resiliency perspective, not least client-death
> (although we could detect that, it would start to get a touch messy).
>
> We could move all this to the client and have, say, a
> PeriodicPublisher which spawns a separate thread. That way, if the
> publishing client dies, the flow stops. The user ends up tied to our
> client library, but since the alternatives are language/framework
> dependent anyway that's not a huge cost. The main downside I see to
> this approach is that we'd need to reimplement this in each client,
> whereas the header or exchange routes avoid that.
>
> The other cost is that at least one extra thread gets spawned. I
> guess, in the Java client at least, we might be able to tie it into
> the dispatcher somehow but that could get hairy..
>
> Re-reading my original mail I wasn't entirely clear about the
> use-case: this was someone who basically wanted to publish the same
> message every N minutes. Think cron with a really basic client that
> sends a message in a static file. the deliver-after stuff was their
> top level stab at doing it in AMQP.
>

So - I'm tempted to say this is not really a Message Queuing use case...  I
can see that the deliver-after type thing could be generally useful
(although you'd want to make it a per-queue option)...  Having messages come
back from the dead periodically seems... odd... and having something that
periodically generates new messages is essentially wishing to write an
application inside the broker... 1-0 would define this as some sort of
service node... Even in 0-8/0-9/0-10 I would be more tempted to write this
as a custom queue than a custom exchange.

In terms of consumption, are you expecting multiple consumers to share the
processing of these messages or is there just one consumer?  In the one
consumer case you could use my LVQ implementation (when I check it in
properly) to read a message which details the periodicity on startup and
then just spawns a thread in the client to perform the task at the
periodicity required... Listening for updates to the periodicity in case it
is changed.

If no-one is listening do you care about having a persistent copy of the
message...?  if no-one is listening for 2xperiod do you expect them to
process 2 separate messages on startup or just the one?  In the latter case
I could buy the case for this being some sort of exchange in 0-8/0-9/0-10
speak

Anyways I'm tempted to implement the deliver-after thing as a general option
on queues...

Cheers,
Rob




> - Aidan
> --
> Apache Qpid - AMQP, JMS, other messaging love http://qpid.apache.org
> "A witty saying proves nothing" - Voltaire
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message