incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Neumeyer (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (S4-5) Prototype app builder
Date Sun, 18 Dec 2011 22:04:30 GMT

    [ https://issues.apache.org/jira/browse/S4-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171944#comment-13171944
] 

Leo Neumeyer edited comment on S4-5 at 12/18/11 10:02 PM:
----------------------------------------------------------

I pushed a new version that seems to be working. I created a new Counter example using the
Fluent API. The maker class tries to infer all the required info and if it cant, it exits
with a message. The main ambiguity happens when a PE has more than one stream field with the
same EventType. In that case, the user must provide the field name using the method withField().
Anyway, implementing and debugging was quite a bit of work. Hope this is useful! The Fluent
API is completely independent and optional. Should be a good example for others who may want
to write a DSL. Doing a DSL in Java is quite cumbersome. 

This is what the example looks like:

{code:title=S4 Fluent API|borderStyle=dashed}
    @Override
    public void configure() {

        /* PE that prints counts to console. */
        PEMaker printPE = addPE(PrintPE.class).asSingleton();

        /* PEs that count events by user, gender, and age. */
        PEMaker userCountPE = addPE(CounterPE.class);
        userCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);

        PEMaker genderCountPE = addPE(CounterPE.class);
        genderCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);

        PEMaker ageCountPE = addPE(CounterPE.class);
        ageCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);

        /* PE that generates random events. */
        generateUserEventPE = addPE(GenerateUserEventPE.class).asSingleton();
        generateUserEventPE.addTimer().withDuration(1, TimeUnit.MILLISECONDS);

        /* Create the application graph. */
        ageCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
        genderCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
        userCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);

        generateUserEventPE.emit(UserEvent.class).onKey(new AgeKeyFinder()).to(ageCountPE);
        generateUserEventPE.emit(UserEvent.class).onKey(new GenderKeyFinder()).to(genderCountPE);
        generateUserEventPE.emit(UserEvent.class).onKey(new UserIDKeyFinder()).to(userCountPE);
    }
{code} 

I made a few additional changes:

* By convention stream fields in PEs must be arrays: Stream<EventType>[]. This convention
helps standardize and build tools.
* Added helper method emit(<? extends Event> event, Stream<? extends Event>[]
streams) in ProcessingElement to avoid having to use a for loop in each PE implementation.
Simply use emit(someOutEvent, someStreamArray).
* Removed the SingletonPE class because it was complicating the API. Instead I added a method
setSingleton() that creates an eagerly singleton in the app. That is, the PE instance is created
and initialized just before the app starts and before events arrive. 
* I improved initialization coordination.
* Started to use the Guava Precondition class because it makes the code cleaner. Please look
at ProcessingElement and use a similar pattern.
* Fixed some bugs.

TODO:

* Expressive language to specify key finder in the DSL.
* More extensive unit testing.


                
      was (Author: leoneu):
    I pushed a new version that seems to be working. I created a new Counter example using
the Fluent API. The maker class tries to infer all the required info and if it cant, it exits
with a message. The main ambiguity happens when a PE has more than one stream field with the
same EventType. In that case, the user must provide the field name using the method withField().
Anyway, implementing and debugging was quite a bit of work. Hope this is useful! The Fluent
API is completely independent and optional. Should be a good example for others who may want
to write a DSL. Doing a DSL in Java is quite cumbersome. 

This is what the example looks like:

{code:title=S4 Fluent API|borderStyle=dashed}
    @Override
    public void configure() {

        /* PE that prints counts to console. */
        PEMaker printPE = addPE(PrintPE.class).asSingleton();

        /* PEs that count events by user, gender, and age. */
        PEMaker userCountPE = addPE(CounterPE.class);
        userCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);

        PEMaker genderCountPE = addPE(CounterPE.class);
        genderCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);

        PEMaker ageCountPE = addPE(CounterPE.class);
        ageCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);

        /* PE that generates random events. */
        generateUserEventPE = addPE(GenerateUserEventPE.class).asSingleton();
        generateUserEventPE.addTimer().withDuration(1, TimeUnit.MILLISECONDS);

        /* Create the application graph. */
        ageCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
        genderCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
        userCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);

        generateUserEventPE.emit(UserEvent.class).onKey(new AgeKeyFinder()).to(ageCountPE);
        generateUserEventPE.emit(UserEvent.class).onKey(new GenderKeyFinder()).to(genderCountPE);
        generateUserEventPE.emit(UserEvent.class).onKey(new UserIDKeyFinder()).to(userCountPE);
    }
{code} 

I made a few additional changes:

* Removed the SingletonPE class because it was complicating the API. Instead I added a method
setSingleton() that creates an eagerly singleton in the app. That is, the PE instance is created
and initialized just before the app starts and before events arrive. 
* I improved initialization coordination.
* Started to use the Guava Precondition class because it makes the code cleaner. Please look
at ProcessingElement and use a similar pattern.
* Fixed some bugs.

TODO:

* Expressive language to specify key finder in the DSL.
* More extensive unit testing.


                  
> Prototype app builder
> ---------------------
>
>                 Key: S4-5
>                 URL: https://issues.apache.org/jira/browse/S4-5
>             Project: Apache S4
>          Issue Type: New Feature
>    Affects Versions: 0.5
>            Reporter: Leo Neumeyer
>            Priority: Minor
>             Fix For: 0.5
>
>
> Extend Guice AbstractModule to create a simple application builder layer. The idea is
to evaluate if we can use Guice to do all the wiring including PEs and Streams and Apps.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message