incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Rohr <rohr.ch...@gmail.com>
Subject Re: reexamining combine
Date Wed, 12 Nov 2014 17:59:55 GMT
I'm all for getting things "right" but would this change affect
getting 0.2.4 out the door?  I think we should focus on finishing this
release up.

Chris

On Wed, Nov 12, 2014 at 9:23 AM, Aaron McCurry <amccurry@gmail.com> wrote:
> No it would an accumulator not a combiner/reducer.  The command would have
> to create an accumulator that would have to be able to accumulate results
> as well as another accumulator instance.  We could the accumulate other
> instances optional, if implemented the accumulator could run on the shard
> servers.  If not implemented, all the accumulation would have to occur on
> the controller.
>
> Aaron
>
> On Wed, Nov 12, 2014 at 9:18 AM, Garrett Barton <garrett.barton@gmail.com>
> wrote:
>
>> Would that be something similar to the Combiner paradigm in MR?
>>
>> On Wed, Nov 12, 2014 at 9:03 AM, Aaron McCurry <amccurry@gmail.com> wrote:
>>
>> > On Tue, Nov 11, 2014 at 3:56 PM, Tim Williams <williamstw@gmail.com>
>> > wrote:
>> >
>> > > Thinking more about the current model, I'm a little concerned the
>> > > current abstraction leaks an internal optimization. From the
>> > > perspective of a command implementor the server combine is just an
>> > > internal optimization?  IOW, from a command perspective, it seems that
>> > > we should be concerned about tables and shards and let
>> > > 'server'-related things be plumbing underneath?
>> > >
>> > > We obviously want to well-define how exactly combine will be called
>> > > (ie. for a given shard result, only ever once).   I dunno, it smells a
>> > > little funny right now, but then again, I know this method-smithing
>> > > can be exhausting, just trying to get a feel for what others think?
>> > > Anyone think it's worth going another round on this?
>> > >
>> >
>> > I hear you on the abstraction/method sigs being a little weird for shard
>> > server combines vs controller combines.  I agree that I would like to let
>> > the commands only have to deal with shard and table results.  However I
>> do
>> > not want to remove an optimization path just for the sake of a cleaner
>> api.
>> >
>> > I suppose we could change a bunch of the logic (again :-) ) and make the
>> > combiner logic be an accumulator that can be serialized across shard and
>> > controller servers.  I haven't really thought this through but that may
>> > allow us to make a single logic path for handling results.  Thoughts?
>> >
>> > Aaron
>> >
>> >
>> > >
>> > > Thanks,
>> > > --tim
>> > >
>> >
>>

Mime
View raw message