incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: more app loading
Date Mon, 24 Oct 2011 20:16:54 GMT
Agree with you. Waiting for JDK 7 is a good idea.

On Mon, Oct 24, 2011 at 11:53 AM, Leo Neumeyer <leoneumeyer@gmail.com>wrote:

> Yes, we look a bit at osgi but seems overkill for what we need. I can
> understand our simple implementation and seems to work for now. A more
> complex framework will require more time and is not even clear that is the
> right solution.  Next JDK will support modular loading so it is probably a
> good idea to wait if we can. Let's see if this works for what we need and we
> can revisit later. However, if you have a good alternative to suggest
> alternatives, please do so.
>
> Our requirements are very constrained because we only need to load
> subclasses of App. This seems to simplify the problem quite a bit. Because
> apps have their own CL, we can easily de-reference the loaded app and have
> the CL + loaded classes get GCed. This seems to satisfies our
> loading/unloading of apps requirement. For inter-app communication we only
> use the EventSource API.
>
> -leo
>
> On Oct 24, 2011, at 11:35 AM, kishore g wrote:
>
> > I know you have tried lot of alternatives for dynamic app/module
> deployment.
> > Just wanted to confirm if we tried Apache Felix.
> >
> > On Mon, Oct 24, 2011 at 11:00 AM, Leo Neumeyer <leoneumeyer@gmail.com
> >wrote:
> >
> >> Nice!
> >>
> >> any thoughts on what will it take to include the dependent jars in the
> s4r
> >> package?
> >>
> >> talk tomorrow,
> >> -leo
> >>
> >> On Oct 24, 2011, at 10:29 AM, Matthieu Morel wrote:
> >>
> >>> Hi,
> >>>
> >>> I had a look into that, and the solution boils down to using the same
> >>> classloader for classes loaded from s4r archives and for classes
> >> dynamically
> >>> generated.
> >>> The JarLoader needs a small modification to handle that.
> >>> It is now possible to load several apps concurrently, including apps
> from
> >>> the s4-example subproject.
> >>>
> >>> I'll share my changes tomorrow so we can discuss and see what the next
> >> steps
> >>> are.
> >>>
> >>> Matthieu
> >>>
> >>>
> >>> On Sun, Oct 23, 2011 at 4:53 AM, Leo Neumeyer <leoneumeyer@gmail.com>
> >> wrote:
> >>>
> >>>> I made some progress getting the app to load. The problem was a
> >>>> missing no-args constructor to get the newInstance method to work.
> >>>> (Apps will be required to have a no-args constructor.) Once the app
> >>>> starts, there is an exception because the CounterPE is not found. It
> >>>> looks like a conflict between dynamic app loading and dynamic
> >>>> dispatching. More debugging needed,  here is the trace:
> >>>>
> >>>> comm.module=org.apache.s4.comm.Module
> >>>> s4.logger_level=TRACE
> >>>> comm.queue_emmiter_size=8000
> >>>> comm.queue_listener_size=8000
> >>>> cluster.hosts=localhost
> >>>> cluster.ports=5077
> >>>> cluster.lock_dir=/tmp
> >>>> cluster.isCluster=true
> >>>> 19:43:53.281 [main] TRACE org.apache.s4.core.Server - Read:
> >>>>
> >>>>
> >>
> file:/Users/leo/Projects/s4project/s4-piper/subprojects/s4-core/bin/apps/CounterExample.s4r
> >>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.MyApp, resolveIt: true
> >>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: org.apache.s4.core.App,
> >>>> resolveIt: true
> >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: org.apache.s4.core.KeyFinder,
> >>>> resolveIt: true
> >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.lang.Throwable,
> >>>> resolveIt: true
> >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> java.lang.InterruptedException, resolveIt: true
> >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.303 [main] INFO  org.apache.s4.comm.topology.Cluster - Added
> >>>> cluster node: localhost:5077
> >>>> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Host
> >>>> Name: 10.0.1.97
> >>>> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> >>>> Partition available: true
> >>>> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> >>>> Partition acquired by PID:16610 HOST:Leos-MacBook-Air.local  Lock File
> >>>> location: /tmp/s4-0.lock
> >>>> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> >>>> Acquire partition:success.
> >>>> 19:43:53.354 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.PrintPE, resolveIt: true
> >>>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.core.ProcessingElement, resolveIt: true
> >>>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.lang.Object, resolveIt:
> >>>> true
> >>>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: org.apache.s4.base.Event,
> >>>> resolveIt: true
> >>>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.CountKeyFinder, resolveIt: true
> >>>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.util.List, resolveIt:
> >>>> true
> >>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.CounterPE, resolveIt: true
> >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.CountEvent, resolveIt: true
> >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: org.apache.s4.core.Stream,
> >>>> resolveIt: true
> >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.407 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.util.concurrent.TimeUnit,
> >>>> resolveIt: true
> >>>> 19:43:53.408 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.413 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.UserIDKeyFinder, resolveIt: true
> >>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.GenderKeyFinder, resolveIt: true
> >>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.AgeKeyFinder, resolveIt: true
> >>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.GenerateUserEventPE, resolveIt: true
> >>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.core.SingletonPE, resolveIt: true
> >>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className:
> >>>> org.apache.s4.example.counter.UserEvent, resolveIt: true
> >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >>>> system class.
> >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning newly loaded class.
> >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.lang.String, resolveIt:
> >>>> true
> >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.util.Random, resolveIt:
> >>>> true
> >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.util.ArrayList,
> >>>> resolveIt: true
> >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.lang.Character,
> >>>> resolveIt: true
> >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.lang.Integer, resolveIt:
> >>>> true
> >>>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.423 [User Stream] TRACE org.apache.s4.core.ProcessingElement
> >>>> - OnCreateInternal
> >>>> 19:43:53.423 [Gender Stream] TRACE
> >>>> org.apache.s4.core.ProcessingElement - OnCreateInternal
> >>>> 19:43:53.423 [Age Stream] TRACE org.apache.s4.core.ProcessingElement
-
> >>>> OnCreateInternal
> >>>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.lang.Thread, resolveIt:
> >>>> true
> >>>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:53.430 [Gender Stream] TRACE
> >>>> org.apache.s4.core.ProcessingElement - Num PE instances: 0.
> >>>> 19:43:53.430 [User Stream] TRACE org.apache.s4.core.ProcessingElement
> >>>> - Num PE instances: 0.
> >>>> 19:43:53.430 [Age Stream] TRACE org.apache.s4.core.ProcessingElement
-
> >>>> Num PE instances: 0.
> >>>> Exception in thread "Age Stream" java.lang.NoClassDefFoundError:
> >>>> org/apache/s4/example/counter/CounterPE
> >>>>      at OverloadDispatcher847.dispatchEvent(Unknown Source)
> >>>>      at
> >>>>
> >>
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
> >>>>      at org.apache.s4.core.Stream.run(Stream.java:272)
> >>>>      at java.lang.Thread.run(Thread.java:680)
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> org.apache.s4.example.counter.CounterPE
> >>>>      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>>      at java.security.AccessController.doPrivileged(Native Method)
> >>>>      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>      ... 4 more
> >>>> Exception in thread "Gender Stream" java.lang.NoClassDefFoundError:
> >>>> org/apache/s4/example/counter/CounterPE
> >>>>      at OverloadDispatcher3022.dispatchEvent(Unknown Source)
> >>>>      at
> >>>>
> >>
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
> >>>>      at org.apache.s4.core.Stream.run(Stream.java:272)
> >>>>      at java.lang.Thread.run(Thread.java:680)
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> org.apache.s4.example.counter.CounterPE
> >>>>      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>>      at java.security.AccessController.doPrivileged(Native Method)
> >>>>      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>      ... 4 more
> >>>> Exception in thread "User Stream" java.lang.NoClassDefFoundError:
> >>>> org/apache/s4/example/counter/CounterPE
> >>>>      at OverloadDispatcher2958.dispatchEvent(Unknown Source)
> >>>>      at
> >>>>
> >>
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
> >>>>      at org.apache.s4.core.Stream.run(Stream.java:272)
> >>>>      at java.lang.Thread.run(Thread.java:680)
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> org.apache.s4.example.counter.CounterPE
> >>>>      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>>      at java.security.AccessController.doPrivileged(Native Method)
> >>>>      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>      ... 4 more
> >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.lang.System, resolveIt:
> >>>> true
> >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> MultiClassLoader loadClass - className: java.io.PrintStream,
> >>>> resolveIt: true
> >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >>>> Returning system class (in CLASSPATH).
> >>>> Done. Closing...
> >>>> Bye.
> >>>> 19:43:58.430 [Age Count Stream] INFO  org.apache.s4.core.Stream -
> >>>> Closing stream Age Count Stream.
> >>>> 19:43:58.430 [Gender Count Stream] INFO  org.apache.s4.core.Stream -
> >>>> Closing stream Gender Count Stream.
> >>>> 19:43:58.430 [User Count Stream] INFO  org.apache.s4.core.Stream -
> >>>> Closing stream User Count Stream.
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Oct 18, 2011 at 11:00 PM, Leo Neumeyer <leoneumeyer@gmail.com
> >
> >>>> wrote:
> >>>>> Matthieu, please check my *modules* branch.
> >>>>> I packaged the counter example in a *.s4r file (a jar with a
> different
> >>>>> extension name). Here is how:
> >>>>> https://github.com/leoneu/s4-piper/blob/modules/NOTES.md
> >>>>> I fixed the code so we can read the jar as a resource from the
> >> classpath.
> >>>>> The HelloApp example works fine but there is an exception when trying
> >> to
> >>>>> load the counter app. The MyApp class should load as is, I think.
>  You
> >>>> will
> >>>>> probably find the bug by the time I wake up :-) The good side of
> being
> >> 9
> >>>>> time zones apart.
> >>>>> -leo
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> -leo
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message