incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: more app loading
Date Mon, 24 Oct 2011 18:35:00 GMT
I know you have tried lot of alternatives for dynamic app/module deployment.
Just wanted to confirm if we tried Apache Felix.

On Mon, Oct 24, 2011 at 11:00 AM, Leo Neumeyer <leoneumeyer@gmail.com>wrote:

> Nice!
>
> any thoughts on what will it take to include the dependent jars in the s4r
> package?
>
> talk tomorrow,
> -leo
>
> On Oct 24, 2011, at 10:29 AM, Matthieu Morel wrote:
>
> > Hi,
> >
> > I had a look into that, and the solution boils down to using the same
> > classloader for classes loaded from s4r archives and for classes
> dynamically
> > generated.
> > The JarLoader needs a small modification to handle that.
> > It is now possible to load several apps concurrently, including apps from
> > the s4-example subproject.
> >
> > I'll share my changes tomorrow so we can discuss and see what the next
> steps
> > are.
> >
> > Matthieu
> >
> >
> > On Sun, Oct 23, 2011 at 4:53 AM, Leo Neumeyer <leoneumeyer@gmail.com>
> wrote:
> >
> >> I made some progress getting the app to load. The problem was a
> >> missing no-args constructor to get the newInstance method to work.
> >> (Apps will be required to have a no-args constructor.) Once the app
> >> starts, there is an exception because the CounterPE is not found. It
> >> looks like a conflict between dynamic app loading and dynamic
> >> dispatching. More debugging needed,  here is the trace:
> >>
> >> comm.module=org.apache.s4.comm.Module
> >> s4.logger_level=TRACE
> >> comm.queue_emmiter_size=8000
> >> comm.queue_listener_size=8000
> >> cluster.hosts=localhost
> >> cluster.ports=5077
> >> cluster.lock_dir=/tmp
> >> cluster.isCluster=true
> >> 19:43:53.281 [main] TRACE org.apache.s4.core.Server - Read:
> >>
> >>
> file:/Users/leo/Projects/s4project/s4-piper/subprojects/s4-core/bin/apps/CounterExample.s4r
> >> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.MyApp, resolveIt: true
> >> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: org.apache.s4.core.App,
> >> resolveIt: true
> >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: org.apache.s4.core.KeyFinder,
> >> resolveIt: true
> >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.lang.Throwable,
> >> resolveIt: true
> >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> java.lang.InterruptedException, resolveIt: true
> >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.303 [main] INFO  org.apache.s4.comm.topology.Cluster - Added
> >> cluster node: localhost:5077
> >> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Host
> >> Name: 10.0.1.97
> >> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> >> Partition available: true
> >> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> >> Partition acquired by PID:16610 HOST:Leos-MacBook-Air.local  Lock File
> >> location: /tmp/s4-0.lock
> >> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> >> Acquire partition:success.
> >> 19:43:53.354 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.PrintPE, resolveIt: true
> >> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.core.ProcessingElement, resolveIt: true
> >> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.lang.Object, resolveIt:
> >> true
> >> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: org.apache.s4.base.Event,
> >> resolveIt: true
> >> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.CountKeyFinder, resolveIt: true
> >> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.util.List, resolveIt:
> >> true
> >> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.CounterPE, resolveIt: true
> >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.CountEvent, resolveIt: true
> >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: org.apache.s4.core.Stream,
> >> resolveIt: true
> >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.407 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.util.concurrent.TimeUnit,
> >> resolveIt: true
> >> 19:43:53.408 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.413 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.UserIDKeyFinder, resolveIt: true
> >> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.GenderKeyFinder, resolveIt: true
> >> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.AgeKeyFinder, resolveIt: true
> >> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.GenerateUserEventPE, resolveIt: true
> >> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.core.SingletonPE, resolveIt: true
> >> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className:
> >> org.apache.s4.example.counter.UserEvent, resolveIt: true
> >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> >> system class.
> >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning newly loaded class.
> >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.lang.String, resolveIt:
> >> true
> >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.util.Random, resolveIt:
> >> true
> >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.util.ArrayList,
> >> resolveIt: true
> >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.lang.Character,
> >> resolveIt: true
> >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.lang.Integer, resolveIt:
> >> true
> >> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.423 [User Stream] TRACE org.apache.s4.core.ProcessingElement
> >> - OnCreateInternal
> >> 19:43:53.423 [Gender Stream] TRACE
> >> org.apache.s4.core.ProcessingElement - OnCreateInternal
> >> 19:43:53.423 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
> >> OnCreateInternal
> >> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.lang.Thread, resolveIt:
> >> true
> >> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:53.430 [Gender Stream] TRACE
> >> org.apache.s4.core.ProcessingElement - Num PE instances: 0.
> >> 19:43:53.430 [User Stream] TRACE org.apache.s4.core.ProcessingElement
> >> - Num PE instances: 0.
> >> 19:43:53.430 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
> >> Num PE instances: 0.
> >> Exception in thread "Age Stream" java.lang.NoClassDefFoundError:
> >> org/apache/s4/example/counter/CounterPE
> >>       at OverloadDispatcher847.dispatchEvent(Unknown Source)
> >>       at
> >>
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
> >>       at org.apache.s4.core.Stream.run(Stream.java:272)
> >>       at java.lang.Thread.run(Thread.java:680)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.s4.example.counter.CounterPE
> >>       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>       at java.security.AccessController.doPrivileged(Native Method)
> >>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>       ... 4 more
> >> Exception in thread "Gender Stream" java.lang.NoClassDefFoundError:
> >> org/apache/s4/example/counter/CounterPE
> >>       at OverloadDispatcher3022.dispatchEvent(Unknown Source)
> >>       at
> >>
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
> >>       at org.apache.s4.core.Stream.run(Stream.java:272)
> >>       at java.lang.Thread.run(Thread.java:680)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.s4.example.counter.CounterPE
> >>       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>       at java.security.AccessController.doPrivileged(Native Method)
> >>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>       ... 4 more
> >> Exception in thread "User Stream" java.lang.NoClassDefFoundError:
> >> org/apache/s4/example/counter/CounterPE
> >>       at OverloadDispatcher2958.dispatchEvent(Unknown Source)
> >>       at
> >>
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
> >>       at org.apache.s4.core.Stream.run(Stream.java:272)
> >>       at java.lang.Thread.run(Thread.java:680)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.s4.example.counter.CounterPE
> >>       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>       at java.security.AccessController.doPrivileged(Native Method)
> >>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>       ... 4 more
> >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.lang.System, resolveIt:
> >> true
> >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> MultiClassLoader loadClass - className: java.io.PrintStream,
> >> resolveIt: true
> >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> >> Returning system class (in CLASSPATH).
> >> Done. Closing...
> >> Bye.
> >> 19:43:58.430 [Age Count Stream] INFO  org.apache.s4.core.Stream -
> >> Closing stream Age Count Stream.
> >> 19:43:58.430 [Gender Count Stream] INFO  org.apache.s4.core.Stream -
> >> Closing stream Gender Count Stream.
> >> 19:43:58.430 [User Count Stream] INFO  org.apache.s4.core.Stream -
> >> Closing stream User Count Stream.
> >>
> >>
> >>
> >> On Tue, Oct 18, 2011 at 11:00 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> >> wrote:
> >>> Matthieu, please check my *modules* branch.
> >>> I packaged the counter example in a *.s4r file (a jar with a different
> >>> extension name). Here is how:
> >>> https://github.com/leoneu/s4-piper/blob/modules/NOTES.md
> >>> I fixed the code so we can read the jar as a resource from the
> classpath.
> >>> The HelloApp example works fine but there is an exception when trying
> to
> >>> load the counter app. The MyApp class should load as is, I think.  You
> >> will
> >>> probably find the bug by the time I wake up :-) The good side of being
> 9
> >>> time zones apart.
> >>> -leo
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> -leo
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message