incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: reexamining combine
Date Wed, 12 Nov 2014 14:23:16 GMT
No it would an accumulator not a combiner/reducer.  The command would have
to create an accumulator that would have to be able to accumulate results
as well as another accumulator instance.  We could the accumulate other
instances optional, if implemented the accumulator could run on the shard
servers.  If not implemented, all the accumulation would have to occur on
the controller.

Aaron

On Wed, Nov 12, 2014 at 9:18 AM, Garrett Barton <garrett.barton@gmail.com>
wrote:

> Would that be something similar to the Combiner paradigm in MR?
>
> On Wed, Nov 12, 2014 at 9:03 AM, Aaron McCurry <amccurry@gmail.com> wrote:
>
> > On Tue, Nov 11, 2014 at 3:56 PM, Tim Williams <williamstw@gmail.com>
> > wrote:
> >
> > > Thinking more about the current model, I'm a little concerned the
> > > current abstraction leaks an internal optimization. From the
> > > perspective of a command implementor the server combine is just an
> > > internal optimization?  IOW, from a command perspective, it seems that
> > > we should be concerned about tables and shards and let
> > > 'server'-related things be plumbing underneath?
> > >
> > > We obviously want to well-define how exactly combine will be called
> > > (ie. for a given shard result, only ever once).   I dunno, it smells a
> > > little funny right now, but then again, I know this method-smithing
> > > can be exhausting, just trying to get a feel for what others think?
> > > Anyone think it's worth going another round on this?
> > >
> >
> > I hear you on the abstraction/method sigs being a little weird for shard
> > server combines vs controller combines.  I agree that I would like to let
> > the commands only have to deal with shard and table results.  However I
> do
> > not want to remove an optimization path just for the sake of a cleaner
> api.
> >
> > I suppose we could change a bunch of the logic (again :-) ) and make the
> > combiner logic be an accumulator that can be serialized across shard and
> > controller servers.  I haven't really thought this through but that may
> > allow us to make a single logic path for handling results.  Thoughts?
> >
> > Aaron
> >
> >
> > >
> > > Thanks,
> > > --tim
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message