incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From adam wojtuniak <>
Subject Re: OSGI for S4
Date Thu, 13 Oct 2011 10:23:22 GMT
Hi Leo,

We can't avoid class loading issue even if we want only to load apps
dynamically. Every app is going to have dependencies to s4 (plus s4
dependencies) and also can happen that app can have their own dependencies,
all these dependencies must be visible to app classloader. One solution is
to expose all these dependencies as a system packages then they will be
visible to all bundles in the osgi container but that is a hack which are
going to lead to maintenance nightmare (every time is new dependency it must
be added to system package). Class.forName is another problem, for example
kryo lib is loading classes which in the container aren't visible to kryo
until kryo will be bundlized with DynamicImport-Package: *. These problems
can't be avoided and is no matter what solution is used (JBossModules or
Felix etc.). One solution is to find proper bundlized version of some s4
dependencies or wrap them using and put for
some of them  DynamicImport-Package: *.


On Wed, Oct 12, 2011 at 12:53 AM, Leo Neumeyer <>wrote:

> Hi Adam,
> very nice.
> We are only interested in loading apps dynamically. Is it possible in OSGI
> to just use it for the apps. The rest of the jars can be loaded at start
> time in the same way we do it now.
> Anyway, I will take a look. Can you post your comments to the dev list so
> everyone can see it and comment?
> s4-dev@incubator.apa**
> Thanks!
> -leo
> On Oct 11, 2011, at 4:41 PM, adam wojtuniak wrote:
> Hi Leo,
> Thanks for description.
> I've implemented ideas which I sent you before. Now I can load the apps
> using embedded osgi container but there is still class loading issue
> (com.esotericsoftware.kryo.SerializationException: Unable to find class:
> io.s4.example.counter.UserEvent). To be able to run s4 I had to put plenty
> system packages in configuration file for felix.
> Around 15 jars defined in libraries are not osgi bundles (don't have
> proper manifest file). The biggest issue are dependencies and classloading
> (for example Kryo is trying to load UserEvent class but that class is not
> visible for him so Class.forName is not going to work).I will try to fix
> these problems, in the meantime you can have a look at my repo
> I added task createRepo, it
> will create bundles directory with all dependencies and project jars. When
> you launch Main class from s4-core, it launches embedded felix which will
> auto deploy all bundles located in bundles directory. In the console you
> will see felix gogo shell (
> ).
> Cheers,
> Adam
> On Sat, Oct 8, 2011 at 5:56 PM, Leo Neumeyer <>wrote:
>> Responding to this:
>> > Maybe you can write me some small description of s4 concept.
>> Here is the very high level s4-piper (which is quite different than s4
>> v0.3). We expect the next version of S4 to be based on s4-piper. This
>> is work in progress!
>> API:
>> * Event: all messages sent between PEs inherit from Event.
>> * ProcessingElement: The unit that processes an event.
>> * Stream: Streams are the edges between PEs. A source pushes events
>> into streams and streams push events into target PEs. Streams are
>> associated with a specific Key.
>> * Key: A value derived from the content of an event. Could be the
>> value of a field,  a combination of fields, or a more complex
>> transformation of the fields. Each PE instance is associated with a
>> unique key value. Keys can also be used to map PEs to partitions in a
>> cluster.
>> * KeyFinder: A helper class that Implements the actual logic used by the
>> Key.
>> * App: A unit that encapsulates a complex task carried out by a graph
>> comprised of PEs and Streams. The App has a reference to all the PE
>> and Stream objects associated with the graph. A deployment may have
>> several Apps that may communicate with each other using EventSource.
>> Apps are associated with owners and units of work. A deployment may
>> include Apps that are or are not related to each other. Apps can also
>> be used to implement security controls. By default, PEs and Streams
>> are not exported outside an App.
>> * Streamable: classes that accept S4 events must implement streamable,
>> currently: Stream and EventSource.
>> * EventSource: use it to export events outside of an App. Any app in
>> the cluster can potentially subscribe to an event source at run time.
>> Some internal classes:
>> * Main: starts a node.
>> * Server: the server instance in a node.
>> * Sender: send an event to its destination in a cluster.
>> * Receiver: receive an event from somewhere in the cluster.
>> Stories:
>> PEs do all the application processing. PE use a prototype pattern
>> which is the key to the S4 design. The first PE object is called a PE
>> prototype which is configured for a specific task but will never be
>> used for actual processing. Events sent to PEs come with a specific
>> key. For example, key=user, key_value=joe. A PE instance, is
>> associated with a specific key, In this case the PE prototype will
>> check if PE for (user, joe) exists. If not, it will clone itself and
>> create a PE instance for (user, joe). The S4 framework guarantees that
>> all future events of the same type sent to a given PE prototype will
>> always arrive at the same PE instance. In this way, we can now do
>> computation for joe. The PE instance may emit an output event sent to
>> another prototype with another key. An App is a graph of PEs and
>> Streams.
>> PEs put events into Stream{s).
>> Stream(s) put events into PEs.
>> Stream(s) are configured for a specific PE target and key finder.
>> To send a user event with field "user" and value "joe" to a UserPE we
>> do the following:
>> - Define UserEvent that includes "user" field.
>> - Define a KeyFinder that extracts the value of field  "user" (will
>> return "joe" in the example.)
>> - Define UserPE that accepts events of type UserEvent and does
>> something with it, say count.
>> - Define a Stream whose target is UserPE configured with the KeyFinder
>> that gets the value of "user".
>> - Now put an object of type UserEvent into the stream and it will get
>> to the unique UserPE instance for (user, joe). The instance could be
>> anywhere in the cluster.
>> Once a server is started, we expect it to run indefinitely until there
>> is a failure or we need to upgrade the framework software or we change
>> the cluster configuration. In S4, the cluster is symmetric, that is,
>> all the nodes are identical, with the same code base and the same
>> configuration.
>> Applications are loaded dynamically after the cluster is started. Most
>> apps need an EventSource to receive events. App must be able to find
>> EventSource and subscribe to it during initialization.
>> When an app is unloaded, it will close its EventSource(s). This in
>> turn, will close all the streams that are subscribed. The apps that
>> own the streams are responsible for taking action once their
>> dependencies are gone.
>> -leo
>> On Sat, Oct 8, 2011 at 8:37 AM, Leo Neumeyer <>
>> wrote:
>> > [sending to the new s4-dev-subscribe list!]
>> > Thank you Adam, let me read the links you sent to understand this
>> better.
>> > BTW. I reorganized the project into subprojects to make it easier to
>> > separate examples from the framework.
>> > To implement inter-app communication I added EventSource which an app
>> can
>> > use to publish its output to other apps and other apps can subscribe to.
>> We
>> > can also use it to build adaptors to non-s4 events. We can use the osgi
>> > mechanisms for declaring dependencies such as app2 depends on app1 and
>> for
>> > app2 to do: app1.subscribeStream(anApp2Stream) so app2 can get events
>> from
>> > app1. Unless an app uses an EventSource, its events will not be
>> exported.
>> > This will create good isolation and make it easier to add a security
>> layer
>> > around apps. Both Stream and EventSource implement Streamable.
>> > My code is here until we set up the apache
>> > svn.
>> > thanks!
>> > -leo
>> >
>> > On Oct 8, 2011, at 5:03 AM, adam wojtuniak wrote:
>> >
>> > Hi Leo,
>> > Yes I have a few ideas but I need couple days to implement them.
>> > First, is to build S4 and examples as proper OSGi bundles using gradle
>> osgi
>> > plugin
>> > Its using bnd tool
>> > Then embed osgi container (using standard osgi api)  in Controller
>> instead
>> > of JBoss modules
>> > see
>> .
>> > and configure directory from which felix should auto deploy our
>> > examples
>> .
>> > For dynamic deployment we can use file install
>> > bundle, it
>> will
>> > watch directory from which it loads new bundles.
>> > To configure properly examples we can use
>> > ipojo or
>> declarative
>> > services
>> > they both using service component model and are not intrusive in the
>> code.
>> > That will be enough to load properly the apps.
>> > Later we have to think how we want to use osgi in s4, just to load apps
>> or
>> > properly use service management. Bundles are hiding the internals from
>> other
>> > bundles and should communicate through well defined services. And also I
>> > don't have clear picture how s4 is going to be distributed (have to look
>> > more at Maybe you can write me some small
>> > description of s4 concept.
>> > Cheers,
>> > Adam
>> >
>> >
>> --
>> -leo

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message