incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: Run an application in a distributed enviroment
Date Tue, 07 Aug 2012 08:37:42 GMT
On 8/6/12 5:03 PM, Davide Simoncelli wrote:
> On computer 1:
>
> ./s4 zkServer &
>
> 02:38:31.650 [main] INFO  org.apache.s4.tools.ZKServer - Starting zookeeper server on
port [2181]
> 02:38:31.655 [main] INFO  org.apache.s4.tools.ZKServer - cleaning existing data in [/tmp/tmp/zookeeper/data]
and [/tmp/tmp/zookeeper/log]
> 02:38:31.925 [main] INFO  org.apache.s4.tools.ZKServer - Zookeeper server started on
port [2181]
>
> ./s4 newCluster -c=cluster1 -nbTasks=1 -flp=12000
>
> 02:38:45.357 [main] INFO  org.apache.s4.tools.DefineCluster - preparing new cluster [cluster1]
with [1] node(s)
> 02:38:45.654 [main] INFO  org.apache.s4.tools.DefineCluster - New cluster configuration
uploaded into zookeeper
>
>
> On computer 2 I run the S4 node:
>
>
> ./s4 node -c=cluster1 -zk 172.16.0.1:2181
>
> calling referenced s4 script : /root/s4-piper/incubator-s4/s4
> 16:25:46.986 [main] INFO  org.apache.s4.core.Main - Initializing S4 node with :
> - comm module class [org.apache.s4.comm.DefaultCommModule]
> - comm configuration file [default.s4.comm.properties from classpath]
> - core module class [org.apache.s4.core.DefaultCoreModule]
> - core configuration file[default.s4.core.properties from classpath]
> - extra modules: []
> - inline parameters: []
> 16:25:47.014 [main] DEBUG org.apache.s4.core.Main - Adding named parameters for injection
: [s4.cluster.zk_address=172.16.0.1:2181]
> 16:25:47.678 [main] INFO  org.apache.s4.core.Main - Starting S4 node. This node will
automatically download applications published for the cluster it belongs to
> 16:25:47.880 [main] INFO  o.a.s.comm.topology.AssignmentFromZK - New session:-2881053689315323;
state is : SyncConnected
> 16:25:48.011 [main] INFO  o.a.s.comm.topology.AssignmentFromZK - Successfully acquired
task:Task-0 by computer2
>
> Then on computer 1 I deploy the application (the S4R archive is already uploaded to computer
2):

What do you mean by "the S4R archive is already uploaded to computer 2"?

The way it is intended to work is that there is a single place where you 
place the archive, and this place is accessible from all nodes in the 
cluster. Currently it may be a location on a shared file system, or on a 
web server (accessible through http).


All S4 nodes then receive a notification that there is an application 
available, fetch it, load it and start it.


Can you check the way you are deploying the application?

Thanks,

Matthieu


>
> ./s4 deploy -s4r=/tmp/testApp.s4r -c=cluster1 -appName=testApp
>
>
> And I see the following output on computer 2 that runs the node:
>
>
>
> 16:25:48.089 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.Server
- Loading application [testApp] from file [/tmp/tmp1277962986968632504s4r]
> 16:25:48.090 [ZkClient-EventThread-13-172.16.0.1:2181] WARN  o.a.s4.base.util.S4RLoaderFactory
- s4.tmp.dir not specified, using temporary directory [/tmp/1344263148090-0] for unpacking
S4R. You may want to specify a parent non-temporary directory.
> 16:25:48.090 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.base.util.S4RLoaderFactory
- Unzipping S4R archive in [/tmp/1344263148090-0]
> 16:25:48.098 [main] INFO  org.apache.s4.core.Server - Loading application [testApp] from
file [/tmp/tmp2955192375317617977s4r]
> 16:25:48.098 [main] WARN  o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified,
using temporary directory [/tmp/1344263148098-0] for unpacking S4R. You may want to specify
a parent non-temporary directory.
> 16:25:48.098 [main] INFO  o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in
[/tmp/1344263148098-0]
> 16:25:48.403 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.Server
- App class name is: testpkg.TestApp
> 16:25:48.435 [main] INFO  org.apache.s4.core.Server - App class name is: testpkg.TestApp
> 16:25:48.519 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClusterFromZK
- Changing cluster topology to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer,taskId=Task-0}]}
from null
> 16:25:48.548 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology
to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}
from null
> 16:25:48.584 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClusterFromZK
- Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@24e33e18
> 16:25:48.607 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Adding topology change
listener:org.apache.s4.comm.tcp.TCPEmitter@3bc486f2
> 16:25:48.632 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClustersFromZK
- New session:-2881053689315320
> 16:25:48.640 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClustersFromZK
- New session:-2881053689315319
> 16:25:48.643 [main] INFO  o.a.s4.comm.topology.ClustersFromZK - New session:-2881053689315318
> 16:25:48.650 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClusterFromZK
- Changing cluster topology to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}
from null
> 16:25:48.651 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.Server
- Loaded application from file /tmp/tmp1277962986968632504s4r
> 16:25:48.651 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s.d.DistributedDeploymentManager
- Successfully installed application testApp
> 16:25:48.679 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator
- Dumping generated overload dispatcher class for PE of class [class testpkg.Block3]
> 16:25:48.709 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology
to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}
from null
> 16:25:48.709 [main] INFO  org.apache.s4.core.Server - Loaded application from file /tmp/tmp2955192375317617977s4r
> 16:25:48.709 [main] INFO  o.a.s.d.DistributedDeploymentManager - Successfully installed
application testApp
> 16:25:48.722 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator
- Dumping generated overload dispatcher class for PE of class [class testpkg.Block2]
> 16:25:48.726 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator
- Dumping generated overload dispatcher class for PE of class [class testpkg.Block1]
> 16:25:48.728 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s4.comm.topology.ClustersFromZK
- Adding input stream [RawData] for app [-1] in cluster [cluster1]
> 16:25:48.732 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class testpkg.Block3]
> 16:25:48.737 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class testpkg.Block2]
> 16:25:48.741 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class testpkg.Block1]
> 16:25:48.838 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClustersFromZK
- Detected new stream [RawData]
> 16:25:48.995 [main] DEBUG o.a.s4.comm.topology.ClustersFromZK - Adding input stream [RawData]
for app [-1] in cluster [cluster1]
> 16:25:48.996 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.App -
Init prototype [testpkg.Block3].
> 16:25:48.998 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.App -
Init prototype [testpkg.Block2].
> 16:25:48.998 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.App -
Init prototype [testpkg.Block1].
> 16:25:49.065 [main] INFO  org.apache.s4.core.App - Init prototype [testpkg.Block3].
> 16:25:49.065 [main] INFO  org.apache.s4.core.App - Init prototype [testpkg.Block2].
> 16:25:49.065 [main] INFO  org.apache.s4.core.App - Init prototype [testpkg.Block1].
> 16:31:40.058 [ZkClient-EventThread-13-172.16.0.1:2181] ERROR o.a.s.d.DistributedDeploymentManager
- Application undeployment is not supported yet
> 16:31:40.508 [ZkClient-EventThread-22-172.16.0.1:2181] ERROR o.a.s4.comm.topology.ClusterFromZK
- Zookeeper session expired, possibly due to a network partition for cluster [{ nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}].
This node is considered as dead by Zookeeper. Proceeding to stop this node.
>
>
> Can you help me?  I can start the application and the adapater  on a local cluster (only
one node).
>
> Thank you
>
> Regards
>
> - Davide
>


Mime
View raw message