incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: more app loading
Date Mon, 24 Oct 2011 17:29:50 GMT
Hi,

I had a look into that, and the solution boils down to using the same
classloader for classes loaded from s4r archives and for classes dynamically
generated.
The JarLoader needs a small modification to handle that.
It is now possible to load several apps concurrently, including apps from
the s4-example subproject.

I'll share my changes tomorrow so we can discuss and see what the next steps
are.

Matthieu


On Sun, Oct 23, 2011 at 4:53 AM, Leo Neumeyer <leoneumeyer@gmail.com> wrote:

> I made some progress getting the app to load. The problem was a
> missing no-args constructor to get the newInstance method to work.
> (Apps will be required to have a no-args constructor.) Once the app
> starts, there is an exception because the CounterPE is not found. It
> looks like a conflict between dynamic app loading and dynamic
> dispatching. More debugging needed,  here is the trace:
>
> comm.module=org.apache.s4.comm.Module
> s4.logger_level=TRACE
> comm.queue_emmiter_size=8000
> comm.queue_listener_size=8000
> cluster.hosts=localhost
> cluster.ports=5077
> cluster.lock_dir=/tmp
> cluster.isCluster=true
> 19:43:53.281 [main] TRACE org.apache.s4.core.Server - Read:
>
> file:/Users/leo/Projects/s4project/s4-piper/subprojects/s4-core/bin/apps/CounterExample.s4r
> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.MyApp, resolveIt: true
> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: org.apache.s4.core.App,
> resolveIt: true
> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: org.apache.s4.core.KeyFinder,
> resolveIt: true
> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.lang.Throwable,
> resolveIt: true
> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> java.lang.InterruptedException, resolveIt: true
> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.303 [main] INFO  org.apache.s4.comm.topology.Cluster - Added
> cluster node: localhost:5077
> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Host
> Name: 10.0.1.97
> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> Partition available: true
> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> Partition acquired by PID:16610 HOST:Leos-MacBook-Air.local  Lock File
> location: /tmp/s4-0.lock
> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
> Acquire partition:success.
> 19:43:53.354 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.PrintPE, resolveIt: true
> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.core.ProcessingElement, resolveIt: true
> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.lang.Object, resolveIt:
> true
> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: org.apache.s4.base.Event,
> resolveIt: true
> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.CountKeyFinder, resolveIt: true
> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.util.List, resolveIt:
> true
> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.CounterPE, resolveIt: true
> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.CountEvent, resolveIt: true
> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: org.apache.s4.core.Stream,
> resolveIt: true
> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.407 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.util.concurrent.TimeUnit,
> resolveIt: true
> 19:43:53.408 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.413 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.UserIDKeyFinder, resolveIt: true
> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.GenderKeyFinder, resolveIt: true
> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.AgeKeyFinder, resolveIt: true
> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.GenerateUserEventPE, resolveIt: true
> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.core.SingletonPE, resolveIt: true
> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className:
> org.apache.s4.example.counter.UserEvent, resolveIt: true
> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
> system class.
> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning newly loaded class.
> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.lang.String, resolveIt:
> true
> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.util.Random, resolveIt:
> true
> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.util.ArrayList,
> resolveIt: true
> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.lang.Character,
> resolveIt: true
> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.lang.Integer, resolveIt:
> true
> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.423 [User Stream] TRACE org.apache.s4.core.ProcessingElement
> - OnCreateInternal
> 19:43:53.423 [Gender Stream] TRACE
> org.apache.s4.core.ProcessingElement - OnCreateInternal
> 19:43:53.423 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
> OnCreateInternal
> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.lang.Thread, resolveIt:
> true
> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:53.430 [Gender Stream] TRACE
> org.apache.s4.core.ProcessingElement - Num PE instances: 0.
> 19:43:53.430 [User Stream] TRACE org.apache.s4.core.ProcessingElement
> - Num PE instances: 0.
> 19:43:53.430 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
> Num PE instances: 0.
> Exception in thread "Age Stream" java.lang.NoClassDefFoundError:
> org/apache/s4/example/counter/CounterPE
>        at OverloadDispatcher847.dispatchEvent(Unknown Source)
>        at
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>        at org.apache.s4.core.Stream.run(Stream.java:272)
>        at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.s4.example.counter.CounterPE
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>        ... 4 more
> Exception in thread "Gender Stream" java.lang.NoClassDefFoundError:
> org/apache/s4/example/counter/CounterPE
>        at OverloadDispatcher3022.dispatchEvent(Unknown Source)
>        at
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>        at org.apache.s4.core.Stream.run(Stream.java:272)
>        at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.s4.example.counter.CounterPE
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>        ... 4 more
> Exception in thread "User Stream" java.lang.NoClassDefFoundError:
> org/apache/s4/example/counter/CounterPE
>        at OverloadDispatcher2958.dispatchEvent(Unknown Source)
>        at
> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>        at org.apache.s4.core.Stream.run(Stream.java:272)
>        at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.s4.example.counter.CounterPE
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>        ... 4 more
> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.lang.System, resolveIt:
> true
> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> MultiClassLoader loadClass - className: java.io.PrintStream,
> resolveIt: true
> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
> Returning system class (in CLASSPATH).
> Done. Closing...
> Bye.
> 19:43:58.430 [Age Count Stream] INFO  org.apache.s4.core.Stream -
> Closing stream Age Count Stream.
> 19:43:58.430 [Gender Count Stream] INFO  org.apache.s4.core.Stream -
> Closing stream Gender Count Stream.
> 19:43:58.430 [User Count Stream] INFO  org.apache.s4.core.Stream -
> Closing stream User Count Stream.
>
>
>
> On Tue, Oct 18, 2011 at 11:00 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> wrote:
> > Matthieu, please check my *modules* branch.
> > I packaged the counter example in a *.s4r file (a jar with a different
> > extension name). Here is how:
> > https://github.com/leoneu/s4-piper/blob/modules/NOTES.md
> > I fixed the code so we can read the jar as a resource from the classpath.
> > The HelloApp example works fine but there is an exception when trying to
> > load the counter app. The MyApp class should load as is, I think.  You
> will
> > probably find the bug by the time I wake up :-) The good side of being 9
> > time zones apart.
> > -leo
> >
>
>
>
> --
>
> -leo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message