incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Neumeyer (Commented) (JIRA)" <>
Subject [jira] [Commented] (S4-5) Prototype app builder
Date Sun, 18 Dec 2011 21:42:31 GMT


Leo Neumeyer commented on S4-5:

I pushed a new version that seems to be working. I created a new Counter example using the
Fluent API. The maker class tries to infer all the required info and if it cant, it exits
with a message. The main ambiguity happens when a PE has more than one stream field with the
same EventType. In that case, the user must provide the field name using the method withField().
Anyway, implementing and debugging was quite a bit of work. Hope this is useful! The Fluent
API is completely independent and optional. Should be a good example for others who may want
to write a DSL. Doing a DSL in Java is quite cumbersome. 

This is what the example looks like:

{code:title=S4 Fluent API|borderStyle=dashed}
    public void configure() {

        /* PE that prints counts to console. */
        PEMaker printPE = addPE(PrintPE.class).asSingleton();

        /* PEs that count events by user, gender, and age. */
        PEMaker userCountPE = addPE(CounterPE.class);
        userCountPE.addTrigger().fireOn(Event.class).ifInterval(100l, TimeUnit.MILLISECONDS);

        PEMaker genderCountPE = addPE(CounterPE.class);
        genderCountPE.addTrigger().fireOn(Event.class).ifInterval(100l, TimeUnit.MILLISECONDS);

        PEMaker ageCountPE = addPE(CounterPE.class);
        ageCountPE.addTrigger().fireOn(Event.class).ifInterval(100l, TimeUnit.MILLISECONDS);

        /* PE that generates random events. */
        generateUserEventPE = addPE(GenerateUserEventPE.class).asSingleton();
        generateUserEventPE.addTimer().withDuration(1, TimeUnit.MILLISECONDS);

        /* Create the application graph. */
        ageCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
        genderCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
        userCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);

        generateUserEventPE.emit(UserEvent.class).onKey(new AgeKeyFinder()).to(ageCountPE);
        generateUserEventPE.emit(UserEvent.class).onKey(new GenderKeyFinder()).to(genderCountPE);
        generateUserEventPE.emit(UserEvent.class).onKey(new UserIDKeyFinder()).to(userCountPE);

I made a few additional changes:

* Removed the SingletonPE class because it was complicating the API. Instead I added a method
setSingleton() that creates an eagerly singleton in the app. That is, the PE instance is created
and initialized just before the app starts and before events arrive. 
* I improved initialization coordination.
* Started to use the Guava Precondition class because it makes the code cleaner. Please look
at ProcessingElement and use a similar pattern.
* Fixed some bugs.

> Prototype app builder
> ---------------------
>                 Key: S4-5
>                 URL:
>             Project: Apache S4
>          Issue Type: New Feature
>    Affects Versions: 0.5
>            Reporter: Leo Neumeyer
>            Priority: Minor
>             Fix For: 0.5
> Extend Guice AbstractModule to create a simple application builder layer. The idea is
to evaluate if we can use Guice to do all the wiring including PEs and Streams and Apps.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message