incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: OSGI for S4
Date Thu, 13 Oct 2011 16:26:26 GMT
Does it mean there is no isolation between different modules that are
dynamically loaded. What happens if there are conflicting jars? Can we
control this isolation problematically ?

On Thu, Oct 13, 2011 at 9:13 AM, adam wojtuniak <adamwojtuniak@gmail.com>wrote:

> Hi Leo,
>
> Maybe it is possible with JBossModules, I haven't noticed that they are not
> really OSGi container.
> "JBoss Modules is designed to work with any existing library or application
> without changes, and its simple naming and resolution strategy is what
> makes
> that possible. Unlike OSGi, JBoss Modules does not implement a container;
> rather, it is a thin bootstrap wrapper for executing an application in a
> modular environment. The moment your application takes control, the modular
> environment is ready to load and link modules as needed. Furthermore,
> modules are never loaded (not even for resolution purposes) until required
> by a dependency, meaning that the performance of a modular application
> depends only on the number of modules actually used (and when they are
> used), rather than the total number of modules in the system. And, they may
> be unloaded by the user at any time."
>
> I am going to investigate that what I can do with that solution.
>
> Regards,
> Adam
>
> On Thu, Oct 13, 2011 at 4:44 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> wrote:
>
> > When I tried jboss modules, I didn't have to do any of this. Maybe it
> > is worth investing more times in getting jboss modules to work.
> >
> > I started a jira for this topic here:
> > https://issues.apache.org/jira/browse/S4-4
> >
> > please add to this issue.
> >
> > -leo
> >
> > On Thu, Oct 13, 2011 at 3:23 AM, adam wojtuniak <adamwojtuniak@gmail.com
> >
> > wrote:
> > > Hi Leo,
> > >
> > > We can't avoid class loading issue even if we want only to load apps
> > > dynamically. Every app is going to have dependencies to s4 (plus s4
> > > dependencies) and also can happen that app can have their own
> > dependencies,
> > > all these dependencies must be visible to app classloader. One solution
> > is
> > > to expose all these dependencies as a system packages then they will be
> > > visible to all bundles in the osgi container but that is a hack which
> are
> > > going to lead to maintenance nightmare (every time is new dependency it
> > must
> > > be added to system package). Class.forName is another problem, for
> > example
> > > kryo lib is loading classes which in the container aren't visible to
> kryo
> > > until kryo will be bundlized with DynamicImport-Package: *. These
> > problems
> > > can't be avoided and is no matter what solution is used (JBossModules
> or
> > > Felix etc.). One solution is to find proper bundlized version of some
> s4
> > > dependencies or wrap them using http://www.aqute.biz/Code/Bnd and put
> > for
> > > some of them  DynamicImport-Package: *.
> > >
> > > Cheers,
> > > Adam
> > >
> > > On Wed, Oct 12, 2011 at 12:53 AM, Leo Neumeyer <
> lneumeyer@quantbench.com
> > >wrote:
> > >
> > >> Hi Adam,
> > >>
> > >> very nice.
> > >>
> > >> We are only interested in loading apps dynamically. Is it possible in
> > OSGI
> > >> to just use it for the apps. The rest of the jars can be loaded at
> start
> > >> time in the same way we do it now.
> > >>
> > >> Anyway, I will take a look. Can you post your comments to the dev list
> > so
> > >> everyone can see it and comment?
> > >>
> > >> s4-dev@incubator.apa**che.org
> > >>
> > >> Thanks!
> > >> -leo
> > >>
> > >>
> > >> On Oct 11, 2011, at 4:41 PM, adam wojtuniak wrote:
> > >>
> > >> Hi Leo,
> > >>
> > >> Thanks for description.
> > >> I've implemented ideas which I sent you before. Now I can load the
> apps
> > >> using embedded osgi container but there is still class loading issue
> > >> (com.esotericsoftware.kryo.SerializationException: Unable to find
> class:
> > >> io.s4.example.counter.UserEvent). To be able to run s4 I had to put
> > plenty
> > >> system packages in configuration file for felix.
> > >> Around 15 jars defined in libraries are not osgi bundles (don't have
> > >> proper manifest file). The biggest issue are dependencies and
> > classloading
> > >> (for example Kryo is trying to load UserEvent class but that class is
> > not
> > >> visible for him so Class.forName is not going to work).I will try to
> fix
> > >> these problems, in the meantime you can have a look at my repo
> > >> https://github.com/adamwojtuniak/s4-piper. I added task createRepo,
> it
> > >> will create bundles directory with all dependencies and project jars.
> > When
> > >> you launch Main class from s4-core, it launches embedded felix which
> > will
> > >> auto deploy all bundles located in bundles directory. In the console
> you
> > >> will see felix gogo shell (
> > >> http://www.packtpub.com/article/apache-felix-gogo ).
> > >>
> > >> Cheers,
> > >> Adam
> > >>
> > >> On Sat, Oct 8, 2011 at 5:56 PM, Leo Neumeyer <leoneumeyer@gmail.com
> > >wrote:
> > >>
> > >>> Responding to this:
> > >>>
> > >>> > Maybe you can write me some small description of s4 concept.
> > >>>
> > >>> Here is the very high level s4-piper (which is quite different than
> s4
> > >>> v0.3). We expect the next version of S4 to be based on s4-piper. This
> > >>> is work in progress!
> > >>>
> > >>> API:
> > >>>
> > >>> * Event: all messages sent between PEs inherit from Event.
> > >>> * ProcessingElement: The unit that processes an event.
> > >>> * Stream: Streams are the edges between PEs. A source pushes events
> > >>> into streams and streams push events into target PEs. Streams are
> > >>> associated with a specific Key.
> > >>> * Key: A value derived from the content of an event. Could be the
> > >>> value of a field,  a combination of fields, or a more complex
> > >>> transformation of the fields. Each PE instance is associated with a
> > >>> unique key value. Keys can also be used to map PEs to partitions in
a
> > >>> cluster.
> > >>> * KeyFinder: A helper class that Implements the actual logic used by
> > the
> > >>> Key.
> > >>> * App: A unit that encapsulates a complex task carried out by a graph
> > >>> comprised of PEs and Streams. The App has a reference to all the PE
> > >>> and Stream objects associated with the graph. A deployment may have
> > >>> several Apps that may communicate with each other using EventSource.
> > >>> Apps are associated with owners and units of work. A deployment may
> > >>> include Apps that are or are not related to each other. Apps can also
> > >>> be used to implement security controls. By default, PEs and Streams
> > >>> are not exported outside an App.
> > >>> * Streamable: classes that accept S4 events must implement
> streamable,
> > >>> currently: Stream and EventSource.
> > >>> * EventSource: use it to export events outside of an App. Any app in
> > >>> the cluster can potentially subscribe to an event source at run time.
> > >>>
> > >>> Some internal classes:
> > >>>
> > >>> * Main: starts a node.
> > >>> * Server: the server instance in a node.
> > >>> * Sender: send an event to its destination in a cluster.
> > >>> * Receiver: receive an event from somewhere in the cluster.
> > >>>
> > >>> Stories:
> > >>>
> > >>> PEs do all the application processing. PE use a prototype pattern
> > >>> which is the key to the S4 design. The first PE object is called a
PE
> > >>> prototype which is configured for a specific task but will never be
> > >>> used for actual processing. Events sent to PEs come with a specific
> > >>> key. For example, key=user, key_value=joe. A PE instance, is
> > >>> associated with a specific key, In this case the PE prototype will
> > >>> check if PE for (user, joe) exists. If not, it will clone itself and
> > >>> create a PE instance for (user, joe). The S4 framework guarantees
> that
> > >>> all future events of the same type sent to a given PE prototype will
> > >>> always arrive at the same PE instance. In this way, we can now do
> > >>> computation for joe. The PE instance may emit an output event sent
to
> > >>> another prototype with another key. An App is a graph of PEs and
> > >>> Streams.
> > >>>
> > >>> PEs put events into Stream{s).
> > >>>
> > >>> Stream(s) put events into PEs.
> > >>>
> > >>> Stream(s) are configured for a specific PE target and key finder.
> > >>>
> > >>> To send a user event with field "user" and value "joe" to a UserPE
we
> > >>> do the following:
> > >>>
> > >>> - Define UserEvent that includes "user" field.
> > >>> - Define a KeyFinder that extracts the value of field  "user" (will
> > >>> return "joe" in the example.)
> > >>> - Define UserPE that accepts events of type UserEvent and does
> > >>> something with it, say count.
> > >>> - Define a Stream whose target is UserPE configured with the
> KeyFinder
> > >>> that gets the value of "user".
> > >>> - Now put an object of type UserEvent into the stream and it will get
> > >>> to the unique UserPE instance for (user, joe). The instance could be
> > >>> anywhere in the cluster.
> > >>>
> > >>> Once a server is started, we expect it to run indefinitely until
> there
> > >>> is a failure or we need to upgrade the framework software or we
> change
> > >>> the cluster configuration. In S4, the cluster is symmetric, that is,
> > >>> all the nodes are identical, with the same code base and the same
> > >>> configuration.
> > >>>
> > >>> Applications are loaded dynamically after the cluster is started.
> Most
> > >>> apps need an EventSource to receive events. App must be able to find
> > >>> EventSource and subscribe to it during initialization.
> > >>>
> > >>> When an app is unloaded, it will close its EventSource(s). This in
> > >>> turn, will close all the streams that are subscribed. The apps that
> > >>> own the streams are responsible for taking action once their
> > >>> dependencies are gone.
> > >>>
> > >>> -leo
> > >>>
> > >>>
> > >>> On Sat, Oct 8, 2011 at 8:37 AM, Leo Neumeyer <leoneumeyer@gmail.com>
> > >>> wrote:
> > >>> > [sending to the new s4-dev-subscribe list!]
> > >>> > Thank you Adam, let me read the links you sent to understand this
> > >>> better.
> > >>> > BTW. I reorganized the project into subprojects to make it easier
> to
> > >>> > separate examples from the framework.
> > >>> > To implement inter-app communication I added EventSource which
an
> app
> > >>> can
> > >>> > use to publish its output to other apps and other apps can
> subscribe
> > to.
> > >>> We
> > >>> > can also use it to build adaptors to non-s4 events. We can use
the
> > osgi
> > >>> > mechanisms for declaring dependencies such as app2 depends on
app1
> > and
> > >>> for
> > >>> > app2 to do: app1.subscribeStream(anApp2Stream) so app2 can get
> events
> > >>> from
> > >>> > app1. Unless an app uses an EventSource, its events will not be
> > >>> exported.
> > >>> > This will create good isolation and make it easier to add a
> security
> > >>> layer
> > >>> > around apps. Both Stream and EventSource implement Streamable.
> > >>> > My code is here until we set up the apache
> > >>> > svn. https://github.com/leoneu/s4-piper
> > >>> > thanks!
> > >>> > -leo
> > >>> >
> > >>> > On Oct 8, 2011, at 5:03 AM, adam wojtuniak wrote:
> > >>> >
> > >>> > Hi Leo,
> > >>> > Yes I have a few ideas but I need couple days to implement them.
> > >>> > First, is to build S4 and examples as proper OSGi bundles using
> > gradle
> > >>> osgi
> > >>> > plugin http://gradle.org/osgi_plugin.
> > >>> > Its using bnd tool http://www.aqute.biz/Bnd/Bnd.
> > >>> > Then embed osgi container (using standard osgi api)  in Controller
> > >>> instead
> > >>> > of JBoss modules
> > >>> > see
> > >>>
> >
> http://felix.apache.org/site/apache-felix-framework-launching-and-embedding.html
> > >>> .
> > >>> > and configure directory from which felix should auto deploy our
> > >>> > examples
> > >>>
> >
> http://felix.apache.org/site/apache-felix-framework-usage-documentation.html#ApacheFelixFrameworkUsageDocumentation-autodeploy
> > >>> .
> > >>> > For dynamic deployment we can use file install
> > >>> > bundle http://felix.apache.org/site/apache-felix-file-install.html
> ,
> > it
> > >>> will
> > >>> > watch directory from which it loads new bundles.
> > >>> > To configure properly examples we can use
> > >>> > ipojo http://felix.apache.org/site/apache-felix-ipojo.html or
> > >>> declarative
> > >>> > services
> > >>>
> > http://felix.apache.org/site/apache-felix-service-component-runtime.html
> ,
> > >>> > they both using service component model and are not intrusive
in
> the
> > >>> code.
> > >>> > That will be enough to load properly the apps.
> > >>> > Later we have to think how we want to use osgi in s4, just to
load
> > apps
> > >>> or
> > >>> > properly use service management. Bundles are hiding the internals
> > from
> > >>> other
> > >>> > bundles and should communicate through well defined services.
And
> > also I
> > >>> > don't have clear picture how s4 is going to be distributed (have
to
> > look
> > >>> > more at https://github.com/s4/s4). Maybe you can write me some
> small
> > >>> > description of s4 concept.
> > >>> > Cheers,
> > >>> > Adam
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> -leo
> > >>>
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> > --
> >
> > -leo
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message