incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From adam wojtuniak <adamwojtun...@gmail.com>
Subject Re: OSGI for S4
Date Thu, 13 Oct 2011 16:13:45 GMT
Hi Leo,

Maybe it is possible with JBossModules, I haven't noticed that they are not
really OSGi container.
"JBoss Modules is designed to work with any existing library or application
without changes, and its simple naming and resolution strategy is what makes
that possible. Unlike OSGi, JBoss Modules does not implement a container;
rather, it is a thin bootstrap wrapper for executing an application in a
modular environment. The moment your application takes control, the modular
environment is ready to load and link modules as needed. Furthermore,
modules are never loaded (not even for resolution purposes) until required
by a dependency, meaning that the performance of a modular application
depends only on the number of modules actually used (and when they are
used), rather than the total number of modules in the system. And, they may
be unloaded by the user at any time."

I am going to investigate that what I can do with that solution.

Regards,
Adam

On Thu, Oct 13, 2011 at 4:44 PM, Leo Neumeyer <leoneumeyer@gmail.com> wrote:

> When I tried jboss modules, I didn't have to do any of this. Maybe it
> is worth investing more times in getting jboss modules to work.
>
> I started a jira for this topic here:
> https://issues.apache.org/jira/browse/S4-4
>
> please add to this issue.
>
> -leo
>
> On Thu, Oct 13, 2011 at 3:23 AM, adam wojtuniak <adamwojtuniak@gmail.com>
> wrote:
> > Hi Leo,
> >
> > We can't avoid class loading issue even if we want only to load apps
> > dynamically. Every app is going to have dependencies to s4 (plus s4
> > dependencies) and also can happen that app can have their own
> dependencies,
> > all these dependencies must be visible to app classloader. One solution
> is
> > to expose all these dependencies as a system packages then they will be
> > visible to all bundles in the osgi container but that is a hack which are
> > going to lead to maintenance nightmare (every time is new dependency it
> must
> > be added to system package). Class.forName is another problem, for
> example
> > kryo lib is loading classes which in the container aren't visible to kryo
> > until kryo will be bundlized with DynamicImport-Package: *. These
> problems
> > can't be avoided and is no matter what solution is used (JBossModules or
> > Felix etc.). One solution is to find proper bundlized version of some s4
> > dependencies or wrap them using http://www.aqute.biz/Code/Bnd and put
> for
> > some of them  DynamicImport-Package: *.
> >
> > Cheers,
> > Adam
> >
> > On Wed, Oct 12, 2011 at 12:53 AM, Leo Neumeyer <lneumeyer@quantbench.com
> >wrote:
> >
> >> Hi Adam,
> >>
> >> very nice.
> >>
> >> We are only interested in loading apps dynamically. Is it possible in
> OSGI
> >> to just use it for the apps. The rest of the jars can be loaded at start
> >> time in the same way we do it now.
> >>
> >> Anyway, I will take a look. Can you post your comments to the dev list
> so
> >> everyone can see it and comment?
> >>
> >> s4-dev@incubator.apa**che.org
> >>
> >> Thanks!
> >> -leo
> >>
> >>
> >> On Oct 11, 2011, at 4:41 PM, adam wojtuniak wrote:
> >>
> >> Hi Leo,
> >>
> >> Thanks for description.
> >> I've implemented ideas which I sent you before. Now I can load the apps
> >> using embedded osgi container but there is still class loading issue
> >> (com.esotericsoftware.kryo.SerializationException: Unable to find class:
> >> io.s4.example.counter.UserEvent). To be able to run s4 I had to put
> plenty
> >> system packages in configuration file for felix.
> >> Around 15 jars defined in libraries are not osgi bundles (don't have
> >> proper manifest file). The biggest issue are dependencies and
> classloading
> >> (for example Kryo is trying to load UserEvent class but that class is
> not
> >> visible for him so Class.forName is not going to work).I will try to fix
> >> these problems, in the meantime you can have a look at my repo
> >> https://github.com/adamwojtuniak/s4-piper. I added task createRepo, it
> >> will create bundles directory with all dependencies and project jars.
> When
> >> you launch Main class from s4-core, it launches embedded felix which
> will
> >> auto deploy all bundles located in bundles directory. In the console you
> >> will see felix gogo shell (
> >> http://www.packtpub.com/article/apache-felix-gogo ).
> >>
> >> Cheers,
> >> Adam
> >>
> >> On Sat, Oct 8, 2011 at 5:56 PM, Leo Neumeyer <leoneumeyer@gmail.com
> >wrote:
> >>
> >>> Responding to this:
> >>>
> >>> > Maybe you can write me some small description of s4 concept.
> >>>
> >>> Here is the very high level s4-piper (which is quite different than s4
> >>> v0.3). We expect the next version of S4 to be based on s4-piper. This
> >>> is work in progress!
> >>>
> >>> API:
> >>>
> >>> * Event: all messages sent between PEs inherit from Event.
> >>> * ProcessingElement: The unit that processes an event.
> >>> * Stream: Streams are the edges between PEs. A source pushes events
> >>> into streams and streams push events into target PEs. Streams are
> >>> associated with a specific Key.
> >>> * Key: A value derived from the content of an event. Could be the
> >>> value of a field,  a combination of fields, or a more complex
> >>> transformation of the fields. Each PE instance is associated with a
> >>> unique key value. Keys can also be used to map PEs to partitions in a
> >>> cluster.
> >>> * KeyFinder: A helper class that Implements the actual logic used by
> the
> >>> Key.
> >>> * App: A unit that encapsulates a complex task carried out by a graph
> >>> comprised of PEs and Streams. The App has a reference to all the PE
> >>> and Stream objects associated with the graph. A deployment may have
> >>> several Apps that may communicate with each other using EventSource.
> >>> Apps are associated with owners and units of work. A deployment may
> >>> include Apps that are or are not related to each other. Apps can also
> >>> be used to implement security controls. By default, PEs and Streams
> >>> are not exported outside an App.
> >>> * Streamable: classes that accept S4 events must implement streamable,
> >>> currently: Stream and EventSource.
> >>> * EventSource: use it to export events outside of an App. Any app in
> >>> the cluster can potentially subscribe to an event source at run time.
> >>>
> >>> Some internal classes:
> >>>
> >>> * Main: starts a node.
> >>> * Server: the server instance in a node.
> >>> * Sender: send an event to its destination in a cluster.
> >>> * Receiver: receive an event from somewhere in the cluster.
> >>>
> >>> Stories:
> >>>
> >>> PEs do all the application processing. PE use a prototype pattern
> >>> which is the key to the S4 design. The first PE object is called a PE
> >>> prototype which is configured for a specific task but will never be
> >>> used for actual processing. Events sent to PEs come with a specific
> >>> key. For example, key=user, key_value=joe. A PE instance, is
> >>> associated with a specific key, In this case the PE prototype will
> >>> check if PE for (user, joe) exists. If not, it will clone itself and
> >>> create a PE instance for (user, joe). The S4 framework guarantees that
> >>> all future events of the same type sent to a given PE prototype will
> >>> always arrive at the same PE instance. In this way, we can now do
> >>> computation for joe. The PE instance may emit an output event sent to
> >>> another prototype with another key. An App is a graph of PEs and
> >>> Streams.
> >>>
> >>> PEs put events into Stream{s).
> >>>
> >>> Stream(s) put events into PEs.
> >>>
> >>> Stream(s) are configured for a specific PE target and key finder.
> >>>
> >>> To send a user event with field "user" and value "joe" to a UserPE we
> >>> do the following:
> >>>
> >>> - Define UserEvent that includes "user" field.
> >>> - Define a KeyFinder that extracts the value of field  "user" (will
> >>> return "joe" in the example.)
> >>> - Define UserPE that accepts events of type UserEvent and does
> >>> something with it, say count.
> >>> - Define a Stream whose target is UserPE configured with the KeyFinder
> >>> that gets the value of "user".
> >>> - Now put an object of type UserEvent into the stream and it will get
> >>> to the unique UserPE instance for (user, joe). The instance could be
> >>> anywhere in the cluster.
> >>>
> >>> Once a server is started, we expect it to run indefinitely until there
> >>> is a failure or we need to upgrade the framework software or we change
> >>> the cluster configuration. In S4, the cluster is symmetric, that is,
> >>> all the nodes are identical, with the same code base and the same
> >>> configuration.
> >>>
> >>> Applications are loaded dynamically after the cluster is started. Most
> >>> apps need an EventSource to receive events. App must be able to find
> >>> EventSource and subscribe to it during initialization.
> >>>
> >>> When an app is unloaded, it will close its EventSource(s). This in
> >>> turn, will close all the streams that are subscribed. The apps that
> >>> own the streams are responsible for taking action once their
> >>> dependencies are gone.
> >>>
> >>> -leo
> >>>
> >>>
> >>> On Sat, Oct 8, 2011 at 8:37 AM, Leo Neumeyer <leoneumeyer@gmail.com>
> >>> wrote:
> >>> > [sending to the new s4-dev-subscribe list!]
> >>> > Thank you Adam, let me read the links you sent to understand this
> >>> better.
> >>> > BTW. I reorganized the project into subprojects to make it easier to
> >>> > separate examples from the framework.
> >>> > To implement inter-app communication I added EventSource which an app
> >>> can
> >>> > use to publish its output to other apps and other apps can subscribe
> to.
> >>> We
> >>> > can also use it to build adaptors to non-s4 events. We can use the
> osgi
> >>> > mechanisms for declaring dependencies such as app2 depends on app1
> and
> >>> for
> >>> > app2 to do: app1.subscribeStream(anApp2Stream) so app2 can get events
> >>> from
> >>> > app1. Unless an app uses an EventSource, its events will not be
> >>> exported.
> >>> > This will create good isolation and make it easier to add a security
> >>> layer
> >>> > around apps. Both Stream and EventSource implement Streamable.
> >>> > My code is here until we set up the apache
> >>> > svn. https://github.com/leoneu/s4-piper
> >>> > thanks!
> >>> > -leo
> >>> >
> >>> > On Oct 8, 2011, at 5:03 AM, adam wojtuniak wrote:
> >>> >
> >>> > Hi Leo,
> >>> > Yes I have a few ideas but I need couple days to implement them.
> >>> > First, is to build S4 and examples as proper OSGi bundles using
> gradle
> >>> osgi
> >>> > plugin http://gradle.org/osgi_plugin.
> >>> > Its using bnd tool http://www.aqute.biz/Bnd/Bnd.
> >>> > Then embed osgi container (using standard osgi api)  in Controller
> >>> instead
> >>> > of JBoss modules
> >>> > see
> >>>
> http://felix.apache.org/site/apache-felix-framework-launching-and-embedding.html
> >>> .
> >>> > and configure directory from which felix should auto deploy our
> >>> > examples
> >>>
> http://felix.apache.org/site/apache-felix-framework-usage-documentation.html#ApacheFelixFrameworkUsageDocumentation-autodeploy
> >>> .
> >>> > For dynamic deployment we can use file install
> >>> > bundle http://felix.apache.org/site/apache-felix-file-install.html,
> it
> >>> will
> >>> > watch directory from which it loads new bundles.
> >>> > To configure properly examples we can use
> >>> > ipojo http://felix.apache.org/site/apache-felix-ipojo.html or
> >>> declarative
> >>> > services
> >>>
> http://felix.apache.org/site/apache-felix-service-component-runtime.html,
> >>> > they both using service component model and are not intrusive in the
> >>> code.
> >>> > That will be enough to load properly the apps.
> >>> > Later we have to think how we want to use osgi in s4, just to load
> apps
> >>> or
> >>> > properly use service management. Bundles are hiding the internals
> from
> >>> other
> >>> > bundles and should communicate through well defined services. And
> also I
> >>> > don't have clear picture how s4 is going to be distributed (have to
> look
> >>> > more at https://github.com/s4/s4). Maybe you can write me some small
> >>> > description of s4 concept.
> >>> > Cheers,
> >>> > Adam
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> -leo
> >>>
> >>
> >>
> >>
> >
>
>
>
> --
>
> -leo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message