incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Neumeyer (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (S4-22) Adaptor
Date Sat, 10 Dec 2011 19:41:40 GMT

    [ https://issues.apache.org/jira/browse/S4-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166959#comment-13166959
] 

Leo Neumeyer commented on S4-22:
--------------------------------

For the records, here is the outpur:

comm.module=org.apache.s4.comm.Module
s4.logger_level=TRACE
s4.apps.path=/tmp/s4apps
comm.queue_emmiter_size=8000
comm.queue_listener_size=8000
cluster.hosts=localhost
cluster.ports=5077
cluster.lock_dir=/tmp
cluster.isCluster=true
11:37:04.602 [main] INFO  org.apache.s4.comm.topology.Cluster - Added cluster node: localhost:5077
11:37:04.645 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Host Name: 10.0.1.97
11:37:04.646 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Partition available: true
11:37:04.666 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Partition acquired by PID:41322
HOST:Leo.local  Lock File location: /tmp/s4-0.lock
11:37:04.666 [main] INFO  o.a.s.c.topology.AssignmentFromFile - Acquire partition:success.
11:37:04.676 [main] INFO  org.apache.s4.core.Server - Loading app: /tmp/s4apps/s4-counter-0.0.0-SNAPSHOT.s4r
11:37:04.680 [main] INFO  org.apache.s4.core.Server - App class name is: s4app.ClockApp
11:37:04.680 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: s4app.ClockApp, resolveIt: true
11:37:04.681 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a system class [s4app.ClockApp]
11:37:04.681 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.App, resolveIt: true
11:37:04.683 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.App]
11:37:04.683 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning newly loaded class
[s4app.ClockApp]
11:37:04.684 [main] INFO  org.apache.s4.core.Server - Loading app: /tmp/s4apps/s4-showtime-0.0.0-SNAPSHOT.s4r
11:37:04.685 [main] INFO  org.apache.s4.core.Server - App class name is: s4app.ShowTimeApp
11:37:04.685 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: s4app.ShowTimeApp, resolveIt: true
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a system class [s4app.ShowTimeApp]
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.App, resolveIt: true
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.App]
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning newly loaded class
[s4app.ShowTimeApp]
11:37:04.686 [main] INFO  org.apache.s4.core.Server - Starting app s4app.ClockApp
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: java.lang.System, resolveIt: true
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[java.lang.System]
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: java.io.PrintStream, resolveIt: true
11:37:04.686 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[java.io.PrintStream]
Initing CounterApp...
11:37:04.687 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: s4app.ClockPE, resolveIt: true
11:37:04.687 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a system class [s4app.ClockPE]
11:37:04.687 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.ProcessingElement, resolveIt: true
11:37:04.689 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.ProcessingElement]
11:37:04.689 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning newly loaded class
[s4app.ClockPE]
11:37:04.692 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.slf4j.LoggerFactory, resolveIt: true
11:37:04.693 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.slf4j.LoggerFactory]
11:37:04.695 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.Streamable, resolveIt: true
11:37:04.695 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.Streamable]
11:37:04.708 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class s4app.ClockPE]
11:37:04.714 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: OverloadDispatcher2841, resolveIt: true
11:37:04.716 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a system class [OverloadDispatcher2841]
11:37:04.716 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.gen.OverloadDispatcher, resolveIt: true
11:37:04.716 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.gen.OverloadDispatcher]
11:37:04.716 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: java.lang.Object, resolveIt: true
11:37:04.716 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[java.lang.Object]
11:37:04.716 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning newly loaded class
[OverloadDispatcher2841]
11:37:04.746 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: java.util.concurrent.TimeUnit, resolveIt: true
11:37:04.746 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[java.util.concurrent.TimeUnit]
11:37:04.747 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.EventSource, resolveIt: true
11:37:04.749 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.EventSource]
Starting CounterApp...
11:37:04.751 [main] TRACE org.apache.s4.core.ProcessingElement - OnCreateInternal
11:37:04.757 [main] TRACE org.apache.s4.core.ProcessingElement - Num PE instances: 0.
11:37:04.757 [main] INFO  org.apache.s4.core.Server - Starting app s4app.ShowTimeApp
11:37:04.757 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: java.lang.System, resolveIt: true
11:37:04.757 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[java.lang.System]
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: java.io.PrintStream, resolveIt: true
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[java.io.PrintStream]
Initing ShowTimeApp...
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: s4app.ShowPE, resolveIt: true
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a system class [s4app.ShowPE]
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.ProcessingElement, resolveIt: true
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.ProcessingElement]
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning newly loaded class
[s4app.ShowPE]
11:37:04.758 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.slf4j.LoggerFactory, resolveIt: true
11:37:04.759 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.slf4j.LoggerFactory]
11:37:04.759 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.base.Event, resolveIt: true
11:37:04.759 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.base.Event]
11:37:04.760 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class s4app.ShowPE]
11:37:04.761 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: OverloadDispatcher783, resolveIt: true
11:37:04.761 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a system class [OverloadDispatcher783]
11:37:04.761 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: org.apache.s4.core.gen.OverloadDispatcher, resolveIt: true
11:37:04.761 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[org.apache.s4.core.gen.OverloadDispatcher]
11:37:04.762 [main] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass -
className: java.lang.Object, resolveIt: true
11:37:04.762 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in CLASSPATH)
[java.lang.Object]
11:37:04.762 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Returning newly loaded class
[OverloadDispatcher783]
Starting ShowTimeApp...
11:37:04.763 [main] TRACE org.apache.s4.core.ProcessingElement - OnCreateInternal
11:37:04.763 [main] TRACE org.apache.s4.core.ProcessingElement - Num PE instances: 0.
11:37:04.763 [main] INFO  org.apache.s4.core.Server - Resolving dependencies.
11:37:04.763 [main] INFO  org.apache.s4.core.Server - Resolving dependencies for s4app.ClockApp
11:37:04.764 [main] INFO  org.apache.s4.core.Server - App [s4app.ClockApp] exports event source
[I can give you the time!].
11:37:04.764 [main] INFO  org.apache.s4.core.Server - Resolving dependencies for s4app.ShowTimeApp
11:37:04.764 [main] INFO  org.apache.s4.core.Server - The consumer app is [s4app.ShowTimeApp].
11:37:04.764 [main] INFO  org.apache.s4.core.Server - Subscribing stream [I need the time.]
from app [s4app.ShowTimeApp] to event source.
11:37:04.764 [main] INFO  org.apache.s4.core.EventSource - Subscribing stream: I need the
time. to event source: I can give you the time!.
11:37:04.764 [main] INFO  org.apache.s4.core.Server - Completed applications startup.
11:37:05.749 [Timer-0] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass
- className: org.apache.s4.base.Event, resolveIt: true
11:37:05.749 [Timer-0] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in
CLASSPATH) [org.apache.s4.base.Event]
11:37:05.749 [Timer-0] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass
- className: java.lang.Long, resolveIt: true
11:37:05.749 [Timer-0] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in
CLASSPATH) [java.lang.Long]
11:37:05.752 [Timer-0] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader loadClass
- className: org.slf4j.Logger, resolveIt: true
11:37:05.752 [Timer-0] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system class (in
CLASSPATH) [org.slf4j.Logger]
11:37:05.752 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 0 and time 1323545825749.
11:37:05.761 [I need the time.] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader
loadClass - className: java.lang.Long, resolveIt: true
11:37:05.761 [I need the time.] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system
class (in CLASSPATH) [java.lang.Long]
11:37:05.761 [I need the time.] DEBUG o.a.s4.base.util.MultiClassLoader - MultiClassLoader
loadClass - className: org.slf4j.Logger, resolveIt: true
11:37:05.761 [I need the time.] DEBUG o.a.s4.base.util.MultiClassLoader - Returning system
class (in CLASSPATH) [org.slf4j.Logger]
11:37:05.761 [I need the time.] INFO  s4app.ShowPE - Received event with tick 0 and time 1323545825749.
11:37:06.749 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 1 and time 1323545826749.
11:37:06.749 [I need the time.] INFO  s4app.ShowPE - Received event with tick 1 and time 1323545826749.
11:37:07.750 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 2 and time 1323545827750.
11:37:07.750 [I need the time.] INFO  s4app.ShowPE - Received event with tick 2 and time 1323545827750.
11:37:08.751 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 3 and time 1323545828751.
11:37:08.752 [I need the time.] INFO  s4app.ShowPE - Received event with tick 3 and time 1323545828751.
11:37:09.752 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 4 and time 1323545829752.
11:37:09.752 [I need the time.] INFO  s4app.ShowPE - Received event with tick 4 and time 1323545829752.
11:37:10.753 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 5 and time 1323545830753.
11:37:10.754 [I need the time.] INFO  s4app.ShowPE - Received event with tick 5 and time 1323545830753.
11:37:11.754 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 6 and time 1323545831754.
11:37:11.754 [I need the time.] INFO  s4app.ShowPE - Received event with tick 6 and time 1323545831754.
11:37:12.755 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 7 and time 1323545832755.
11:37:12.755 [I need the time.] INFO  s4app.ShowPE - Received event with tick 7 and time 1323545832755.
11:37:13.757 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 8 and time 1323545833757.
11:37:13.757 [I need the time.] INFO  s4app.ShowPE - Received event with tick 8 and time 1323545833757.
11:37:14.756 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 9 and time 1323545834756.
11:37:14.756 [I need the time.] INFO  s4app.ShowPE - Received event with tick 9 and time 1323545834756.
11:37:15.758 [Timer-0] INFO  s4app.ClockPE - Sending event with tick 10 and time 1323545835757.
11:37:15.758 [I need the time.] INFO  s4app.ShowPE - Received event with tick 10 and time
1323545835757.

                
> Adaptor
> -------
>
>                 Key: S4-22
>                 URL: https://issues.apache.org/jira/browse/S4-22
>             Project: Apache S4
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Leo Neumeyer
>            Assignee: Bruce Robbins
>             Fix For: 0.5
>
>         Attachments: s4-subclusters.pdf
>
>
> Need an adaptor for v0.5
> Idea I posted earlier:
> What do you think of this idea for a simple adaptor:
> - Adaptor extends App
> - Adaptor can send events but not receive (for now)
> - Adaptor is deployed as a regular App to the S4 cluster and as an
> Adaptor type  in a host (separate from the S4 cluster).
> - Adaptor, unlike regular apps, can accept event data (in any format)
> directly, not via comm layer.
> - Input data is transformed into S4 events using a modular approach
> and by providing standard modules such as JSON.
> - Output events are exposed using EventSource and consumed by other
> apps without even knowing that they are Adaptors (only the App type is
> exposed in the cluster).
> - S4 events can be processed locally using PEs and Streams as usual.
> (We kind of need to get a local Sender for the local PEs and a
> standard cluster Sender for the EventSource object.)
> So why this approach?
> The GOOD:
> - Seems to be the least disruptive way to inject external events
> - Apps can easily consume the events in a modular way without any
> dependencies. Getting events from an adaptor or from another app is
> identical.
> - The adaptor would be packaged and deployed to the cluster as if it
> was an App (no incremental cost)
> - The adaptor can do preprocessing using the same programming model
> and can reuse PEs.
> The CHALLENGE:
> -  We need to also deploy the Adaptor in a separate host. On the other
> hand, this is inevitable. At least we use the same approach instead of
> creating a different system.
> -  The Adaptor will need to be integrated with ZK to get the physical addresses.
> -  We need to deal with two senders.
> for later: two-way communication and adapter clusters.
> thoughts?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message