incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Neumeyer <>
Subject Re: OSGI for S4
Date Thu, 13 Oct 2011 15:44:25 GMT
When I tried jboss modules, I didn't have to do any of this. Maybe it
is worth investing more times in getting jboss modules to work.

I started a jira for this topic here:

please add to this issue.


On Thu, Oct 13, 2011 at 3:23 AM, adam wojtuniak <> wrote:
> Hi Leo,
> We can't avoid class loading issue even if we want only to load apps
> dynamically. Every app is going to have dependencies to s4 (plus s4
> dependencies) and also can happen that app can have their own dependencies,
> all these dependencies must be visible to app classloader. One solution is
> to expose all these dependencies as a system packages then they will be
> visible to all bundles in the osgi container but that is a hack which are
> going to lead to maintenance nightmare (every time is new dependency it must
> be added to system package). Class.forName is another problem, for example
> kryo lib is loading classes which in the container aren't visible to kryo
> until kryo will be bundlized with DynamicImport-Package: *. These problems
> can't be avoided and is no matter what solution is used (JBossModules or
> Felix etc.). One solution is to find proper bundlized version of some s4
> dependencies or wrap them using and put for
> some of them  DynamicImport-Package: *.
> Cheers,
> Adam
> On Wed, Oct 12, 2011 at 12:53 AM, Leo Neumeyer <>wrote:
>> Hi Adam,
>> very nice.
>> We are only interested in loading apps dynamically. Is it possible in OSGI
>> to just use it for the apps. The rest of the jars can be loaded at start
>> time in the same way we do it now.
>> Anyway, I will take a look. Can you post your comments to the dev list so
>> everyone can see it and comment?
>> s4-dev@incubator.apa**
>> Thanks!
>> -leo
>> On Oct 11, 2011, at 4:41 PM, adam wojtuniak wrote:
>> Hi Leo,
>> Thanks for description.
>> I've implemented ideas which I sent you before. Now I can load the apps
>> using embedded osgi container but there is still class loading issue
>> (com.esotericsoftware.kryo.SerializationException: Unable to find class:
>> io.s4.example.counter.UserEvent). To be able to run s4 I had to put plenty
>> system packages in configuration file for felix.
>> Around 15 jars defined in libraries are not osgi bundles (don't have
>> proper manifest file). The biggest issue are dependencies and classloading
>> (for example Kryo is trying to load UserEvent class but that class is not
>> visible for him so Class.forName is not going to work).I will try to fix
>> these problems, in the meantime you can have a look at my repo
>> I added task createRepo, it
>> will create bundles directory with all dependencies and project jars. When
>> you launch Main class from s4-core, it launches embedded felix which will
>> auto deploy all bundles located in bundles directory. In the console you
>> will see felix gogo shell (
>> ).
>> Cheers,
>> Adam
>> On Sat, Oct 8, 2011 at 5:56 PM, Leo Neumeyer <>wrote:
>>> Responding to this:
>>> > Maybe you can write me some small description of s4 concept.
>>> Here is the very high level s4-piper (which is quite different than s4
>>> v0.3). We expect the next version of S4 to be based on s4-piper. This
>>> is work in progress!
>>> API:
>>> * Event: all messages sent between PEs inherit from Event.
>>> * ProcessingElement: The unit that processes an event.
>>> * Stream: Streams are the edges between PEs. A source pushes events
>>> into streams and streams push events into target PEs. Streams are
>>> associated with a specific Key.
>>> * Key: A value derived from the content of an event. Could be the
>>> value of a field,  a combination of fields, or a more complex
>>> transformation of the fields. Each PE instance is associated with a
>>> unique key value. Keys can also be used to map PEs to partitions in a
>>> cluster.
>>> * KeyFinder: A helper class that Implements the actual logic used by the
>>> Key.
>>> * App: A unit that encapsulates a complex task carried out by a graph
>>> comprised of PEs and Streams. The App has a reference to all the PE
>>> and Stream objects associated with the graph. A deployment may have
>>> several Apps that may communicate with each other using EventSource.
>>> Apps are associated with owners and units of work. A deployment may
>>> include Apps that are or are not related to each other. Apps can also
>>> be used to implement security controls. By default, PEs and Streams
>>> are not exported outside an App.
>>> * Streamable: classes that accept S4 events must implement streamable,
>>> currently: Stream and EventSource.
>>> * EventSource: use it to export events outside of an App. Any app in
>>> the cluster can potentially subscribe to an event source at run time.
>>> Some internal classes:
>>> * Main: starts a node.
>>> * Server: the server instance in a node.
>>> * Sender: send an event to its destination in a cluster.
>>> * Receiver: receive an event from somewhere in the cluster.
>>> Stories:
>>> PEs do all the application processing. PE use a prototype pattern
>>> which is the key to the S4 design. The first PE object is called a PE
>>> prototype which is configured for a specific task but will never be
>>> used for actual processing. Events sent to PEs come with a specific
>>> key. For example, key=user, key_value=joe. A PE instance, is
>>> associated with a specific key, In this case the PE prototype will
>>> check if PE for (user, joe) exists. If not, it will clone itself and
>>> create a PE instance for (user, joe). The S4 framework guarantees that
>>> all future events of the same type sent to a given PE prototype will
>>> always arrive at the same PE instance. In this way, we can now do
>>> computation for joe. The PE instance may emit an output event sent to
>>> another prototype with another key. An App is a graph of PEs and
>>> Streams.
>>> PEs put events into Stream{s).
>>> Stream(s) put events into PEs.
>>> Stream(s) are configured for a specific PE target and key finder.
>>> To send a user event with field "user" and value "joe" to a UserPE we
>>> do the following:
>>> - Define UserEvent that includes "user" field.
>>> - Define a KeyFinder that extracts the value of field  "user" (will
>>> return "joe" in the example.)
>>> - Define UserPE that accepts events of type UserEvent and does
>>> something with it, say count.
>>> - Define a Stream whose target is UserPE configured with the KeyFinder
>>> that gets the value of "user".
>>> - Now put an object of type UserEvent into the stream and it will get
>>> to the unique UserPE instance for (user, joe). The instance could be
>>> anywhere in the cluster.
>>> Once a server is started, we expect it to run indefinitely until there
>>> is a failure or we need to upgrade the framework software or we change
>>> the cluster configuration. In S4, the cluster is symmetric, that is,
>>> all the nodes are identical, with the same code base and the same
>>> configuration.
>>> Applications are loaded dynamically after the cluster is started. Most
>>> apps need an EventSource to receive events. App must be able to find
>>> EventSource and subscribe to it during initialization.
>>> When an app is unloaded, it will close its EventSource(s). This in
>>> turn, will close all the streams that are subscribed. The apps that
>>> own the streams are responsible for taking action once their
>>> dependencies are gone.
>>> -leo
>>> On Sat, Oct 8, 2011 at 8:37 AM, Leo Neumeyer <>
>>> wrote:
>>> > [sending to the new s4-dev-subscribe list!]
>>> > Thank you Adam, let me read the links you sent to understand this
>>> better.
>>> > BTW. I reorganized the project into subprojects to make it easier to
>>> > separate examples from the framework.
>>> > To implement inter-app communication I added EventSource which an app
>>> can
>>> > use to publish its output to other apps and other apps can subscribe to.
>>> We
>>> > can also use it to build adaptors to non-s4 events. We can use the osgi
>>> > mechanisms for declaring dependencies such as app2 depends on app1 and
>>> for
>>> > app2 to do: app1.subscribeStream(anApp2Stream) so app2 can get events
>>> from
>>> > app1. Unless an app uses an EventSource, its events will not be
>>> exported.
>>> > This will create good isolation and make it easier to add a security
>>> layer
>>> > around apps. Both Stream and EventSource implement Streamable.
>>> > My code is here until we set up the apache
>>> > svn.
>>> > thanks!
>>> > -leo
>>> >
>>> > On Oct 8, 2011, at 5:03 AM, adam wojtuniak wrote:
>>> >
>>> > Hi Leo,
>>> > Yes I have a few ideas but I need couple days to implement them.
>>> > First, is to build S4 and examples as proper OSGi bundles using gradle
>>> osgi
>>> > plugin
>>> > Its using bnd tool
>>> > Then embed osgi container (using standard osgi api)  in Controller
>>> instead
>>> > of JBoss modules
>>> > see
>>> .
>>> > and configure directory from which felix should auto deploy our
>>> > examples
>>> .
>>> > For dynamic deployment we can use file install
>>> > bundle, it
>>> will
>>> > watch directory from which it loads new bundles.
>>> > To configure properly examples we can use
>>> > ipojo or
>>> declarative
>>> > services
>>> > they both using service component model and are not intrusive in the
>>> code.
>>> > That will be enough to load properly the apps.
>>> > Later we have to think how we want to use osgi in s4, just to load apps
>>> or
>>> > properly use service management. Bundles are hiding the internals from
>>> other
>>> > bundles and should communicate through well defined services. And also I
>>> > don't have clear picture how s4 is going to be distributed (have to look
>>> > more at Maybe you can write me some small
>>> > description of s4 concept.
>>> > Cheers,
>>> > Adam
>>> >
>>> >
>>> --
>>> -leo



View raw message