incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <>
Subject Re: new queue capability
Date Thu, 27 Feb 2014 16:10:09 GMT
What if we provide an implementation of the QueueReader concept that does
what you are discussing.  That way in more extreme cases when the user is
forced into implementing the lower level api (perhaps for performance) they
can still do it, but for the normal case the partitioning (and other
difficult issues) are handled by the controllers.

I could see adding an enqueueMutate call to the controllers that pushes the
mutates to the correct buckets for the user.  At the same time we could
allow each of the controllers to pull from an external and push the mutates
to the correct buckets for the shards.  I could see a couple of different
ways of handling this.

However I do agree that right now there is too much burden on the user for
the 95% case.  We should make this simpler.


On Thu, Feb 27, 2014 at 10:07 AM, Tim Williams <> wrote:

> I've been playing around with the new QueueReader stuff and I'm
> starting to believe it's at the wrong level of abstraction - in the
> shard context - for a user.
> Between having to know about the BlurPartioner and handling all the
> failure nuances, I'm thinking a much friendlier approach would be to
> have the client implement a single message pump that Blur take's from
> and handles.
> Maybe on startup the Controllers compete for the lead QueueReader
> position, create it from the TableContext and run with it?  The user
> would still need to deal with  Controller failures but that seems
> easier to reason about then shard failures.
> The way it's crafted right now, the user seems burdened with a lot of
> the hard problems that Blur otherwise solves.  Obviously, it trades
> off a high burden for one of the controllers.
> Thoughts?
> --tim

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message