incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Neumeyer (Commented) (JIRA)" <>
Subject [jira] [Commented] (S4-22) Adaptor
Date Fri, 11 Nov 2011 19:36:52 GMT


Leo Neumeyer commented on S4-22:

Bruce suggested a further generalization of this idea:

Instead of one cluster, S4 will support many clusters that can be configured based on requirements.
For example, a typical use case is to have to pre-process raw inbound events at high speed.
This will require adapting the event format, filtering based on logic, etc. We can dedicate
an S4 cluster to this function. The generic S4 cluster would be a separate one. In this case
we have an adaptor (processes a single stream) and a general cluster (processes all streams
allowing combinations across streams.) If we make all node clusters symmetric (they all have
identical code running), then it is easy to do intercluster communication. We just need to
support multiple senders, the API would be unchanged. The only complexity is in configuring
the clusters and senders and configuring inter-cluster streams. Because an adaptor is also
an App, there is very little we need to do to support adaptors.

All inter-app dependencies need to be established during deployment using run-time tables
via ZK. I think that it will be easier to start building a simple admin web app to do this.
The alternative would be a command line interface but not sure it's necessary. The web app
would be used to configure the cluster(s), assign names, load/unload apps, establish dependencies,
 monitor Apps/PEs, etc. SInce Play Framework seems to be the latest hot thing, maybe we can
give it a try, any volunteers?

> Adaptor
> -------
>                 Key: S4-22
>                 URL:
>             Project: Apache S4
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Leo Neumeyer
>            Assignee: Bruce Robbins
> Need an adaptor for v0.5
> Idea I posted earlier:
> What do you think of this idea for a simple adaptor:
> - Adaptor extends App
> - Adaptor can send events but not receive (for now)
> - Adaptor is deployed as a regular App to the S4 cluster and as an
> Adaptor type  in a host (separate from the S4 cluster).
> - Adaptor, unlike regular apps, can accept event data (in any format)
> directly, not via comm layer.
> - Input data is transformed into S4 events using a modular approach
> and by providing standard modules such as JSON.
> - Output events are exposed using EventSource and consumed by other
> apps without even knowing that they are Adaptors (only the App type is
> exposed in the cluster).
> - S4 events can be processed locally using PEs and Streams as usual.
> (We kind of need to get a local Sender for the local PEs and a
> standard cluster Sender for the EventSource object.)
> So why this approach?
> The GOOD:
> - Seems to be the least disruptive way to inject external events
> - Apps can easily consume the events in a modular way without any
> dependencies. Getting events from an adaptor or from another app is
> identical.
> - The adaptor would be packaged and deployed to the cluster as if it
> was an App (no incremental cost)
> - The adaptor can do preprocessing using the same programming model
> and can reuse PEs.
> -  We need to also deploy the Adaptor in a separate host. On the other
> hand, this is inevitable. At least we use the same approach instead of
> creating a different system.
> -  The Adaptor will need to be integrated with ZK to get the physical addresses.
> -  We need to deal with two senders.
> for later: two-way communication and adapter clusters.
> thoughts?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message