incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Neumeyer <leoneume...@gmail.com>
Subject Re: more app loading
Date Mon, 24 Oct 2011 18:53:55 GMT
Yes, we look a bit at osgi but seems overkill for what we need. I can understand our simple
implementation and seems to work for now. A more complex framework will require more time
and is not even clear that is the right solution.  Next JDK will support modular loading so
it is probably a good idea to wait if we can. Let's see if this works for what we need and
we can revisit later. However, if you have a good alternative to suggest alternatives, please
do so.

Our requirements are very constrained because we only need to load subclasses of App. This
seems to simplify the problem quite a bit. Because apps have their own CL, we can easily de-reference
the loaded app and have the CL + loaded classes get GCed. This seems to satisfies our loading/unloading
of apps requirement. For inter-app communication we only use the EventSource API. 

-leo

On Oct 24, 2011, at 11:35 AM, kishore g wrote:

> I know you have tried lot of alternatives for dynamic app/module deployment.
> Just wanted to confirm if we tried Apache Felix.
> 
> On Mon, Oct 24, 2011 at 11:00 AM, Leo Neumeyer <leoneumeyer@gmail.com>wrote:
> 
>> Nice!
>> 
>> any thoughts on what will it take to include the dependent jars in the s4r
>> package?
>> 
>> talk tomorrow,
>> -leo
>> 
>> On Oct 24, 2011, at 10:29 AM, Matthieu Morel wrote:
>> 
>>> Hi,
>>> 
>>> I had a look into that, and the solution boils down to using the same
>>> classloader for classes loaded from s4r archives and for classes
>> dynamically
>>> generated.
>>> The JarLoader needs a small modification to handle that.
>>> It is now possible to load several apps concurrently, including apps from
>>> the s4-example subproject.
>>> 
>>> I'll share my changes tomorrow so we can discuss and see what the next
>> steps
>>> are.
>>> 
>>> Matthieu
>>> 
>>> 
>>> On Sun, Oct 23, 2011 at 4:53 AM, Leo Neumeyer <leoneumeyer@gmail.com>
>> wrote:
>>> 
>>>> I made some progress getting the app to load. The problem was a
>>>> missing no-args constructor to get the newInstance method to work.
>>>> (Apps will be required to have a no-args constructor.) Once the app
>>>> starts, there is an exception because the CounterPE is not found. It
>>>> looks like a conflict between dynamic app loading and dynamic
>>>> dispatching. More debugging needed,  here is the trace:
>>>> 
>>>> comm.module=org.apache.s4.comm.Module
>>>> s4.logger_level=TRACE
>>>> comm.queue_emmiter_size=8000
>>>> comm.queue_listener_size=8000
>>>> cluster.hosts=localhost
>>>> cluster.ports=5077
>>>> cluster.lock_dir=/tmp
>>>> cluster.isCluster=true
>>>> 19:43:53.281 [main] TRACE org.apache.s4.core.Server - Read:
>>>> 
>>>> 
>> file:/Users/leo/Projects/s4project/s4-piper/subprojects/s4-core/bin/apps/CounterExample.s4r
>>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.MyApp, resolveIt: true
>>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: org.apache.s4.core.App,
>>>> resolveIt: true
>>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: org.apache.s4.core.KeyFinder,
>>>> resolveIt: true
>>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.lang.Throwable,
>>>> resolveIt: true
>>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> java.lang.InterruptedException, resolveIt: true
>>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.303 [main] INFO  org.apache.s4.comm.topology.Cluster - Added
>>>> cluster node: localhost:5077
>>>> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Host
>>>> Name: 10.0.1.97
>>>> 19:43:53.336 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
>>>> Partition available: true
>>>> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
>>>> Partition acquired by PID:16610 HOST:Leos-MacBook-Air.local  Lock File
>>>> location: /tmp/s4-0.lock
>>>> 19:43:53.352 [main] INFO  o.a.s.c.topology.AssignmentFromFile -
>>>> Acquire partition:success.
>>>> 19:43:53.354 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.PrintPE, resolveIt: true
>>>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.core.ProcessingElement, resolveIt: true
>>>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.lang.Object, resolveIt:
>>>> true
>>>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: org.apache.s4.base.Event,
>>>> resolveIt: true
>>>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.CountKeyFinder, resolveIt: true
>>>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.util.List, resolveIt:
>>>> true
>>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.CounterPE, resolveIt: true
>>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.CountEvent, resolveIt: true
>>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: org.apache.s4.core.Stream,
>>>> resolveIt: true
>>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.407 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.util.concurrent.TimeUnit,
>>>> resolveIt: true
>>>> 19:43:53.408 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.413 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.UserIDKeyFinder, resolveIt: true
>>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.GenderKeyFinder, resolveIt: true
>>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.AgeKeyFinder, resolveIt: true
>>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.GenerateUserEventPE, resolveIt: true
>>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.core.SingletonPE, resolveIt: true
>>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className:
>>>> org.apache.s4.example.counter.UserEvent, resolveIt: true
>>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a
>>>> system class.
>>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning newly loaded class.
>>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.lang.String, resolveIt:
>>>> true
>>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.util.Random, resolveIt:
>>>> true
>>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.util.ArrayList,
>>>> resolveIt: true
>>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.lang.Character,
>>>> resolveIt: true
>>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.lang.Integer, resolveIt:
>>>> true
>>>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.423 [User Stream] TRACE org.apache.s4.core.ProcessingElement
>>>> - OnCreateInternal
>>>> 19:43:53.423 [Gender Stream] TRACE
>>>> org.apache.s4.core.ProcessingElement - OnCreateInternal
>>>> 19:43:53.423 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
>>>> OnCreateInternal
>>>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.lang.Thread, resolveIt:
>>>> true
>>>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:53.430 [Gender Stream] TRACE
>>>> org.apache.s4.core.ProcessingElement - Num PE instances: 0.
>>>> 19:43:53.430 [User Stream] TRACE org.apache.s4.core.ProcessingElement
>>>> - Num PE instances: 0.
>>>> 19:43:53.430 [Age Stream] TRACE org.apache.s4.core.ProcessingElement -
>>>> Num PE instances: 0.
>>>> Exception in thread "Age Stream" java.lang.NoClassDefFoundError:
>>>> org/apache/s4/example/counter/CounterPE
>>>>      at OverloadDispatcher847.dispatchEvent(Unknown Source)
>>>>      at
>>>> 
>> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>>>>      at org.apache.s4.core.Stream.run(Stream.java:272)
>>>>      at java.lang.Thread.run(Thread.java:680)
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.s4.example.counter.CounterPE
>>>>      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>      at java.security.AccessController.doPrivileged(Native Method)
>>>>      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>      ... 4 more
>>>> Exception in thread "Gender Stream" java.lang.NoClassDefFoundError:
>>>> org/apache/s4/example/counter/CounterPE
>>>>      at OverloadDispatcher3022.dispatchEvent(Unknown Source)
>>>>      at
>>>> 
>> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>>>>      at org.apache.s4.core.Stream.run(Stream.java:272)
>>>>      at java.lang.Thread.run(Thread.java:680)
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.s4.example.counter.CounterPE
>>>>      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>      at java.security.AccessController.doPrivileged(Native Method)
>>>>      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>      ... 4 more
>>>> Exception in thread "User Stream" java.lang.NoClassDefFoundError:
>>>> org/apache/s4/example/counter/CounterPE
>>>>      at OverloadDispatcher2958.dispatchEvent(Unknown Source)
>>>>      at
>>>> 
>> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376)
>>>>      at org.apache.s4.core.Stream.run(Stream.java:272)
>>>>      at java.lang.Thread.run(Thread.java:680)
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.s4.example.counter.CounterPE
>>>>      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>      at java.security.AccessController.doPrivileged(Native Method)
>>>>      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>      ... 4 more
>>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.lang.System, resolveIt:
>>>> true
>>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> MultiClassLoader loadClass - className: java.io.PrintStream,
>>>> resolveIt: true
>>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader -
>>>> Returning system class (in CLASSPATH).
>>>> Done. Closing...
>>>> Bye.
>>>> 19:43:58.430 [Age Count Stream] INFO  org.apache.s4.core.Stream -
>>>> Closing stream Age Count Stream.
>>>> 19:43:58.430 [Gender Count Stream] INFO  org.apache.s4.core.Stream -
>>>> Closing stream Gender Count Stream.
>>>> 19:43:58.430 [User Count Stream] INFO  org.apache.s4.core.Stream -
>>>> Closing stream User Count Stream.
>>>> 
>>>> 
>>>> 
>>>> On Tue, Oct 18, 2011 at 11:00 PM, Leo Neumeyer <leoneumeyer@gmail.com>
>>>> wrote:
>>>>> Matthieu, please check my *modules* branch.
>>>>> I packaged the counter example in a *.s4r file (a jar with a different
>>>>> extension name). Here is how:
>>>>> https://github.com/leoneu/s4-piper/blob/modules/NOTES.md
>>>>> I fixed the code so we can read the jar as a resource from the
>> classpath.
>>>>> The HelloApp example works fine but there is an exception when trying
>> to
>>>>> load the counter app. The MyApp class should load as is, I think.  You
>>>> will
>>>>> probably find the bug by the time I wake up :-) The good side of being
>> 9
>>>>> time zones apart.
>>>>> -leo
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> -leo
>>>> 
>> 
>> 


Mime
View raw message