incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Neumeyer <>
Subject Re: OSGI for S4
Date Sat, 08 Oct 2011 16:56:34 GMT
Responding to this:

> Maybe you can write me some small description of s4 concept.

Here is the very high level s4-piper (which is quite different than s4
v0.3). We expect the next version of S4 to be based on s4-piper. This
is work in progress!


* Event: all messages sent between PEs inherit from Event.
* ProcessingElement: The unit that processes an event.
* Stream: Streams are the edges between PEs. A source pushes events
into streams and streams push events into target PEs. Streams are
associated with a specific Key.
* Key: A value derived from the content of an event. Could be the
value of a field,  a combination of fields, or a more complex
transformation of the fields. Each PE instance is associated with a
unique key value. Keys can also be used to map PEs to partitions in a
* KeyFinder: A helper class that Implements the actual logic used by the Key.
* App: A unit that encapsulates a complex task carried out by a graph
comprised of PEs and Streams. The App has a reference to all the PE
and Stream objects associated with the graph. A deployment may have
several Apps that may communicate with each other using EventSource.
Apps are associated with owners and units of work. A deployment may
include Apps that are or are not related to each other. Apps can also
be used to implement security controls. By default, PEs and Streams
are not exported outside an App.
* Streamable: classes that accept S4 events must implement streamable,
currently: Stream and EventSource.
* EventSource: use it to export events outside of an App. Any app in
the cluster can potentially subscribe to an event source at run time.

Some internal classes:

* Main: starts a node.
* Server: the server instance in a node.
* Sender: send an event to its destination in a cluster.
* Receiver: receive an event from somewhere in the cluster.


PEs do all the application processing. PE use a prototype pattern
which is the key to the S4 design. The first PE object is called a PE
prototype which is configured for a specific task but will never be
used for actual processing. Events sent to PEs come with a specific
key. For example, key=user, key_value=joe. A PE instance, is
associated with a specific key, In this case the PE prototype will
check if PE for (user, joe) exists. If not, it will clone itself and
create a PE instance for (user, joe). The S4 framework guarantees that
all future events of the same type sent to a given PE prototype will
always arrive at the same PE instance. In this way, we can now do
computation for joe. The PE instance may emit an output event sent to
another prototype with another key. An App is a graph of PEs and

PEs put events into Stream{s).

Stream(s) put events into PEs.

Stream(s) are configured for a specific PE target and key finder.

To send a user event with field "user" and value "joe" to a UserPE we
do the following:

- Define UserEvent that includes "user" field.
- Define a KeyFinder that extracts the value of field  "user" (will
return "joe" in the example.)
- Define UserPE that accepts events of type UserEvent and does
something with it, say count.
- Define a Stream whose target is UserPE configured with the KeyFinder
that gets the value of "user".
- Now put an object of type UserEvent into the stream and it will get
to the unique UserPE instance for (user, joe). The instance could be
anywhere in the cluster.

Once a server is started, we expect it to run indefinitely until there
is a failure or we need to upgrade the framework software or we change
the cluster configuration. In S4, the cluster is symmetric, that is,
all the nodes are identical, with the same code base and the same

Applications are loaded dynamically after the cluster is started. Most
apps need an EventSource to receive events. App must be able to find
EventSource and subscribe to it during initialization.

When an app is unloaded, it will close its EventSource(s). This in
turn, will close all the streams that are subscribed. The apps that
own the streams are responsible for taking action once their
dependencies are gone.


On Sat, Oct 8, 2011 at 8:37 AM, Leo Neumeyer <> wrote:
> [sending to the new s4-dev-subscribe list!]
> Thank you Adam, let me read the links you sent to understand this better.
> BTW. I reorganized the project into subprojects to make it easier to
> separate examples from the framework.
> To implement inter-app communication I added EventSource which an app can
> use to publish its output to other apps and other apps can subscribe to. We
> can also use it to build adaptors to non-s4 events. We can use the osgi
> mechanisms for declaring dependencies such as app2 depends on app1 and for
> app2 to do: app1.subscribeStream(anApp2Stream) so app2 can get events from
> app1. Unless an app uses an EventSource, its events will not be exported.
> This will create good isolation and make it easier to add a security layer
> around apps. Both Stream and EventSource implement Streamable.
> My code is here until we set up the apache
> svn.
> thanks!
> -leo
> On Oct 8, 2011, at 5:03 AM, adam wojtuniak wrote:
> Hi Leo,
> Yes I have a few ideas but I need couple days to implement them.
> First, is to build S4 and examples as proper OSGi bundles using gradle osgi
> plugin
> Its using bnd tool
> Then embed osgi container (using standard osgi api)  in Controller instead
> of JBoss modules
> see
> and configure directory from which felix should auto deploy our
> examples
> For dynamic deployment we can use file install
> bundle, it will
> watch directory from which it loads new bundles.
> To configure properly examples we can use
> ipojo or declarative
> services,
> they both using service component model and are not intrusive in the code.
> That will be enough to load properly the apps.
> Later we have to think how we want to use osgi in s4, just to load apps or
> properly use service management. Bundles are hiding the internals from other
> bundles and should communicate through well defined services. And also I
> don't have clear picture how s4 is going to be distributed (have to look
> more at Maybe you can write me some small
> description of s4 concept.
> Cheers,
> Adam



View raw message