incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Neumeyer <leoneume...@gmail.com>
Subject Re: more app loading
Date Mon, 24 Oct 2011 18:00:34 GMT
Nice! 

any thoughts on what will it take to include the dependent jars in the s4r package? 

talk tomorrow,
-leo

On Oct 24, 2011, at 10:29 AM, Matthieu Morel wrote:

> Hi,
> 
> I had a look into that, and the solution boils down to using the same
> classloader for classes loaded from s4r archives and for classes dynamically
> generated.
> The JarLoader needs a small modification to handle that.
> It is now possible to load several apps concurrently, including apps from
> the s4-example subproject.
> 
> I'll share my changes tomorrow so we can discuss and see what the next steps
> are.
> 
> Matthieu
> 
> 
> On Sun, Oct 23, 2011 at 4:53 AM, Leo Neumeyer <leoneumeyer@gmail.com> wrote:
> 
>> I made some progress getting the app to load. The problem was a
>> missing no-args constructor to get the newInstance method to work.
>> (Apps will be required to have a no-args constructor.) Once the app
>> starts, there is an exception because the CounterPE is not found. It
>> looks like a conflict between dynamic app loading and dynamic
>> dispatching. More debugging needed,  here is the trace:
>> 
>> comm.module=org.apache.s4.comm.Module
>> s4.logger_level=TRACE
>> comm.queue_emmiter_size=8000
>> comm.queue_listener_size=8000
>> cluster.hosts=localhost
>> cluster.ports=5077
>> cluster.lock_dir=/tmp
>> cluster.isCluster=true
>> 19:43:53.281 [main] TRACE org.apache.s4.core.Server - Read:
>> 
>> file:/Users/leo/Projects/s4project/s4-piper/subprojects/s4-core/bin/apps/CounterExample.s4r
>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.MyApp, resolveIt: true
>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: org.apache.s4.core.App,
>> resolveIt: true
>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: org.apache.s4.core.KeyFinder,
>> resolveIt: true
>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.lang.Throwable,
>> resolveIt: true
>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> java.lang.InterruptedException, resolveIt: true
>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.303 [main] INFO  org.apache.s4.comm.topology.Cluster - Added
>> cluster node: localhost:5077
>> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Host
>> Name: 10.0.1.97
>> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
>> Partition available: true
>> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
>> Partition acquired by PID:16610 HOST:Leos-MacBook-Air.local  Lock File
>> location: /tmp/s4-0.lock
>> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
>> Acquire partition:success.
>> 19:43:53.354 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.PrintPE, resolveIt: true
>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.core.ProcessingElement, resolveIt: true
>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.lang.Object, resolveIt:
>> true
>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: org.apache.s4.base.Event,
>> resolveIt: true
>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.CountKeyFinder, resolveIt: true
>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.util.List, resolveIt:
>> true
>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.CounterPE, resolveIt: true
>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.CountEvent, resolveIt: true
>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: org.apache.s4.core.Stream,
>> resolveIt: true
>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.407 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.util.concurrent.TimeUnit,
>> resolveIt: true
>> 19:43:53.408 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.413 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.UserIDKeyFinder, resolveIt: true
>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.GenderKeyFinder, resolveIt: true
>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.AgeKeyFinder, resolveIt: true
>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.GenerateUserEventPE, resolveIt: true
>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.core.SingletonPE, resolveIt: true
>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className:
>> org.apache.s4.example.counter.UserEvent, resolveIt: true
>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>> system class.
>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning newly loaded class.
>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.lang.String, resolveIt:
>> true
>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.util.Random, resolveIt:
>> true
>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.util.ArrayList,
>> resolveIt: true
>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.lang.Character,
>> resolveIt: true
>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.lang.Integer, resolveIt:
>> true
>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.423 [User Stream] TRACE org.apache.s4.core.ProcessingElement
>> - OnCreateInternal
>> 19:43:53.423 [Gender Stream] TRACE
>> org.apache.s4.core.ProcessingElement - OnCreateInternal
>> 19:43:53.423 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
>> OnCreateInternal
>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.lang.Thread, resolveIt:
>> true
>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:53.430 [Gender Stream] TRACE
>> org.apache.s4.core.ProcessingElement - Num PE instances: 0.
>> 19:43:53.430 [User Stream] TRACE org.apache.s4.core.ProcessingElement
>> - Num PE instances: 0.
>> 19:43:53.430 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
>> Num PE instances: 0.
>> Exception in thread "Age Stream" java.lang.NoClassDefFoundError:
>> org/apache/s4/example/counter/CounterPE
>>       at OverloadDispatcher847.dispatchEvent(Unknown Source)
>>       at
>> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>>       at org.apache.s4.core.Stream.run(Stream.java:272)
>>       at java.lang.Thread.run(Thread.java:680)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.s4.example.counter.CounterPE
>>       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>       at java.security.AccessController.doPrivileged(Native Method)
>>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>       ... 4 more
>> Exception in thread "Gender Stream" java.lang.NoClassDefFoundError:
>> org/apache/s4/example/counter/CounterPE
>>       at OverloadDispatcher3022.dispatchEvent(Unknown Source)
>>       at
>> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>>       at org.apache.s4.core.Stream.run(Stream.java:272)
>>       at java.lang.Thread.run(Thread.java:680)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.s4.example.counter.CounterPE
>>       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>       at java.security.AccessController.doPrivileged(Native Method)
>>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>       ... 4 more
>> Exception in thread "User Stream" java.lang.NoClassDefFoundError:
>> org/apache/s4/example/counter/CounterPE
>>       at OverloadDispatcher2958.dispatchEvent(Unknown Source)
>>       at
>> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>>       at org.apache.s4.core.Stream.run(Stream.java:272)
>>       at java.lang.Thread.run(Thread.java:680)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.s4.example.counter.CounterPE
>>       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>       at java.security.AccessController.doPrivileged(Native Method)
>>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>       ... 4 more
>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.lang.System, resolveIt:
>> true
>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> MultiClassLoader loadClass - className: java.io.PrintStream,
>> resolveIt: true
>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>> Returning system class (in CLASSPATH).
>> Done. Closing...
>> Bye.
>> 19:43:58.430 [Age Count Stream] INFO  org.apache.s4.core.Stream -
>> Closing stream Age Count Stream.
>> 19:43:58.430 [Gender Count Stream] INFO  org.apache.s4.core.Stream -
>> Closing stream Gender Count Stream.
>> 19:43:58.430 [User Count Stream] INFO  org.apache.s4.core.Stream -
>> Closing stream User Count Stream.
>> 
>> 
>> 
>> On Tue, Oct 18, 2011 at 11:00 PM, Leo Neumeyer <leoneumeyer@gmail.com>
>> wrote:
>>> Matthieu, please check my *modules* branch.
>>> I packaged the counter example in a *.s4r file (a jar with a different
>>> extension name). Here is how:
>>> https://github.com/leoneu/s4-piper/blob/modules/NOTES.md
>>> I fixed the code so we can read the jar as a resource from the classpath.
>>> The HelloApp example works fine but there is an exception when trying to
>>> load the counter app. The MyApp class should load as is, I think.  You
>> will
>>> probably find the bug by the time I wake up :-) The good side of being 9
>>> time zones apart.
>>> -leo
>>> 
>> 
>> 
>> 
>> --
>> 
>> -leo
>> 


Mime
View raw message