fluo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: [DISCUSS] New API for setting up observers.
Date Fri, 24 Feb 2017 16:25:46 GMT
On Thu, Feb 23, 2017 at 7:03 PM, Christopher <ctubbsii@apache.org> wrote:
> I like the idea, but I am curious how Fluo should handle the case where the
> Observer implementation is different for different workers (either the
> factory provides different Observer implementations on different workers,
> or the workers are running different versions).

This can happen currently, with different implementations of the same
class.  Using YARN to launch workers makes this situation less likely
because YARN will launch all processes with the same code.  Also
commit e211990[1] added code to deal with YARN leaving stragglers
behind as outlined in #482[2].  If not using YARN to launch workers,
then the issue of workers using different versions of code is more
likely to occur.

[1]: https://github.com/apache/incubator-fluo/commit/e211990f0e96b88485cd2e522cbf6032524cd1a3
[2]: https://github.com/apache/incubator-fluo/issues/482

> On Thu, Feb 23, 2017 at 5:36 PM Keith Turner <keith@deenlo.com> wrote:
>> The current way observers are set up in Fluo requires specifying a
>> class for each observers in configuration.  This is cumbersome and
>> prevents using lambdas and anonymous inner classes.  It also makes it
>> hard to follow what a Fluo Application is doing.  This cumbersome way
>> of setting things up propagates forward into higher level constructs
>> in Fluo Recipes making them also cumbersome to use.
>> I think it would be much simpler if the user only had to specify one
>> factory class in configuration that created all observers.  This
>> factory would be free to pair a lambda with a column for an observer.
>> Observed columns would no longer be tightly coupled with a specific
>> class.  I am thinking the factory would look something like the
>> following.
>> interface ObserversFactory {
>>   Map<ObservedColumn, Observer> getObservers(SimpleConfiguration
>> applicationConfig);
>> }
>> A user would implement this interface with a class that creates all of
>> their observers.
>> class UserObserversFactory implements ObserversFactory {
>>   Map<ObservedColumn, Observer> getObservers(SimpleConfiguration
>> appConfig) {
>>     //all of users observers are setup here in code, which is much
>> easier to follow than current way of configuring each observer class
>> individually.
>>      HashMap observers = ...;
>>      observers.put(col1, tx,row,col -> ....); //an observer that's a lambda
>>      ExportQueue.addObserver(observers, appConfig, "queueId", exports
>> -> ....);  //using a lambda to handle exports...
>>   }
>> }
>> The user would configure Fluo to use the above observer factory.
>> I am thinking of trying to implement this for 1.1, but wanted to see
>> if anyone had any input before I started.  If it seems to work well, I
>> was thinking of deprecating the current way of configuring observers.
>> I would update webindex and fluo recipes in parallel to evaluate the
>> changes.
> --
> Christopher

View raw message