incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davide Simoncelli <Davide.Simonce...@neclab.eu>
Subject Run an application in a distributed enviroment
Date Mon, 06 Aug 2012 15:03:43 GMT
On computer 1:

./s4 zkServer &

02:38:31.650 [main] INFO  org.apache.s4.tools.ZKServer - Starting zookeeper server on port
[2181]
02:38:31.655 [main] INFO  org.apache.s4.tools.ZKServer - cleaning existing data in [/tmp/tmp/zookeeper/data]
and [/tmp/tmp/zookeeper/log]
02:38:31.925 [main] INFO  org.apache.s4.tools.ZKServer - Zookeeper server started on port
[2181]

./s4 newCluster -c=cluster1 -nbTasks=1 -flp=12000

02:38:45.357 [main] INFO  org.apache.s4.tools.DefineCluster - preparing new cluster [cluster1]
with [1] node(s)
02:38:45.654 [main] INFO  org.apache.s4.tools.DefineCluster - New cluster configuration uploaded
into zookeeper


On computer 2 I run the S4 node:


./s4 node -c=cluster1 -zk 172.16.0.1:2181

calling referenced s4 script : /root/s4-piper/incubator-s4/s4
16:25:46.986 [main] INFO  org.apache.s4.core.Main - Initializing S4 node with :
- comm module class [org.apache.s4.comm.DefaultCommModule]
- comm configuration file [default.s4.comm.properties from classpath]
- core module class [org.apache.s4.core.DefaultCoreModule]
- core configuration file[default.s4.core.properties from classpath]
- extra modules: []
- inline parameters: []
16:25:47.014 [main] DEBUG org.apache.s4.core.Main - Adding named parameters for injection
: [s4.cluster.zk_address=172.16.0.1:2181]
16:25:47.678 [main] INFO  org.apache.s4.core.Main - Starting S4 node. This node will automatically
download applications published for the cluster it belongs to
16:25:47.880 [main] INFO  o.a.s.comm.topology.AssignmentFromZK - New session:-2881053689315323;
state is : SyncConnected
16:25:48.011 [main] INFO  o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0
by computer2

Then on computer 1 I deploy the application (the S4R archive is already uploaded to computer
2):

./s4 deploy -s4r=/tmp/testApp.s4r -c=cluster1 -appName=testApp


And I see the following output on computer 2 that runs the node:



16:25:48.089 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.Server - Loading
application [testApp] from file [/tmp/tmp1277962986968632504s4r]
16:25:48.090 [ZkClient-EventThread-13-172.16.0.1:2181] WARN  o.a.s4.base.util.S4RLoaderFactory
- s4.tmp.dir not specified, using temporary directory [/tmp/1344263148090-0] for unpacking
S4R. You may want to specify a parent non-temporary directory.
16:25:48.090 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.base.util.S4RLoaderFactory
- Unzipping S4R archive in [/tmp/1344263148090-0]
16:25:48.098 [main] INFO  org.apache.s4.core.Server - Loading application [testApp] from file
[/tmp/tmp2955192375317617977s4r]
16:25:48.098 [main] WARN  o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using
temporary directory [/tmp/1344263148098-0] for unpacking S4R. You may want to specify a parent
non-temporary directory.
16:25:48.098 [main] INFO  o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1344263148098-0]
16:25:48.403 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.Server - App
class name is: testpkg.TestApp
16:25:48.435 [main] INFO  org.apache.s4.core.Server - App class name is: testpkg.TestApp
16:25:48.519 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClusterFromZK
- Changing cluster topology to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer,taskId=Task-0}]}
from null
16:25:48.548 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to
{ nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}
from null
16:25:48.584 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClusterFromZK
- Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@24e33e18
16:25:48.607 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@3bc486f2
16:25:48.632 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClustersFromZK
- New session:-2881053689315320
16:25:48.640 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClustersFromZK
- New session:-2881053689315319
16:25:48.643 [main] INFO  o.a.s4.comm.topology.ClustersFromZK - New session:-2881053689315318
16:25:48.650 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClusterFromZK
- Changing cluster topology to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}
from null
16:25:48.651 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.Server - Loaded
application from file /tmp/tmp1277962986968632504s4r
16:25:48.651 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s.d.DistributedDeploymentManager
- Successfully installed application testApp
16:25:48.679 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator
- Dumping generated overload dispatcher class for PE of class [class testpkg.Block3]
16:25:48.709 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to
{ nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}
from null
16:25:48.709 [main] INFO  org.apache.s4.core.Server - Loaded application from file /tmp/tmp2955192375317617977s4r
16:25:48.709 [main] INFO  o.a.s.d.DistributedDeploymentManager - Successfully installed application
testApp
16:25:48.722 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator
- Dumping generated overload dispatcher class for PE of class [class testpkg.Block2]
16:25:48.726 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator
- Dumping generated overload dispatcher class for PE of class [class testpkg.Block1]
16:25:48.728 [ZkClient-EventThread-13-172.16.0.1:2181] DEBUG o.a.s4.comm.topology.ClustersFromZK
- Adding input stream [RawData] for app [-1] in cluster [cluster1]
16:25:48.732 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class testpkg.Block3]
16:25:48.737 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class testpkg.Block2]
16:25:48.741 [main] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload
dispatcher class for PE of class [class testpkg.Block1]
16:25:48.838 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  o.a.s4.comm.topology.ClustersFromZK
- Detected new stream [RawData]
16:25:48.995 [main] DEBUG o.a.s4.comm.topology.ClustersFromZK - Adding input stream [RawData]
for app [-1] in cluster [cluster1]
16:25:48.996 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.App - Init
prototype [testpkg.Block3].
16:25:48.998 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.App - Init
prototype [testpkg.Block2].
16:25:48.998 [ZkClient-EventThread-13-172.16.0.1:2181] INFO  org.apache.s4.core.App - Init
prototype [testpkg.Block1].
16:25:49.065 [main] INFO  org.apache.s4.core.App - Init prototype [testpkg.Block3].
16:25:49.065 [main] INFO  org.apache.s4.core.App - Init prototype [testpkg.Block2].
16:25:49.065 [main] INFO  org.apache.s4.core.App - Init prototype [testpkg.Block1].
16:31:40.058 [ZkClient-EventThread-13-172.16.0.1:2181] ERROR o.a.s.d.DistributedDeploymentManager
- Application undeployment is not supported yet
16:31:40.508 [ZkClient-EventThread-22-172.16.0.1:2181] ERROR o.a.s4.comm.topology.ClusterFromZK
- Zookeeper session expired, possibly due to a network partition for cluster [{ nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=computer2,taskId=Task-0}]}].
This node is considered as dead by Zookeeper. Proceeding to stop this node.


Can you help me?  I can start the application and the adapater  on a local cluster (only one
node).

Thank you

Regards

- Davide

Mime
View raw message