incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: To deploy an app on multiple machines
Date Wed, 29 Aug 2012 07:58:06 GMT
Hi,

this is expected with the "hello" example adapter, which is written to 
show basic usage only.

As you can see in the code reported in the stack trace 
(HelloInputAdapter, line 42), the code opens a socket on a hardcoded 
port number (15000). This fails is the port is already taken (by another 
"hello" adapter in your case).

It is quite easy to overcome this limitation by passing the port number 
as a parameter (see "Application configuration" on the wiki 
https://cwiki.apache.org/confluence/display/S4/Configuration), or by 
incrementing the port number if already taken, for example.

Hope this helps,

Matthieu




On 8/29/12 4:15 AM, JiHyoun Park wrote:
> Hi Matthieu
>
> With the same source code (the Hello sample), I tried to test a
> 2-adapters case using just one machine.
> But when I deployed the adapter app (HelloInputAdapter), I got the
> following error from one of adapters. (The other adapter was OK.)
>
> ************************************************************************************************************************************
>
>     09:43:14.310 [ZkClient-EventThread-13-localhost:2181] INFO
>     org.apache.s4.core.Server - Loaded application from file
>     /tmp/tmp3145808370992354402s4r
>     09:43:14.310 [ZkClient-EventThread-13-localhost:2181] INFO
>     o.a.s.d.DistributedDeploymentManager - Successfully installed
>     application s4App
>     09:43:14.311 [ZkClient-EventThread-13-localhost:2181] DEBUG
>     o.a.s4.comm.topology.ClustersFromZK - Adding output stream [names]
>     for app [-1] in cluster [cluster2]
>     java.net.BindException: Address already in use
>          at java.net.PlainSocketImpl.socketBind(Native Method)
>          at
>     java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
>          at java.net.ServerSocket.bind(ServerSocket.java:376)
>          at java.net.ServerSocket.<init>(ServerSocket.java:237)
>          at java.net.ServerSocket.<init>(ServerSocket.java:128)
>          at hello.HelloInputAdapter$1.run(HelloInputAdapter.java:42)
>          at java.lang.Thread.run(Thread.java:722)
>
> *******************************************************************************************************************
>
> Are there any instructions to run multiple adapters?
> S4 status looks fine.
>
> ***********************************************************************************************************************
>
>     [root@s4test4 myApp]# ./s4 status
>     calling referenced s4 script : /root/apache-s4-0.5.0-incubating-src/s4
>
>
>     App Status
>     ----------------------------------------------------------------------------------------------------------------------------------
>              Name
>     Cluster                                                  URI
>     ----------------------------------------------------------------------------------------------------------------------------------
>             s4App              cluster2
>     file:/tmp/1346204592699-0/s4App.s4r
>             myApp             cluster1
>     file:/tmp/1346204586930-0/myApp.s4r
>     ----------------------------------------------------------------------------------------------------------------------------------
>
>     Cluster Status
>     ----------------------------------------------------------------------------------------------------------------------------------
>                                                                                     
    Active nodes
>              Name                App           Tasks
>     --------------------------------------------------------------------------------
>                                                         Number    Task
>     id                         Host                         Port
>     ----------------------------------------------------------------------------------------------------------------------------------
>            cluster2             s4App           2         2
>     Task-0                    s4test4                     13000
>
>                                       Task-1
>     s4test4                     13001
>            cluster1             myApp           2         2
>     Task-0                    s4test4                     12000
>
>                          Task-1
>     s4test4                     12001
>     ----------------------------------------------------------------------------------------------------------------------------------
>
>     Stream Status
>     ----------------------------------------------------------------------------------------------------------------------------------
>              Name
>     Producers                                              Consumers
>     ----------------------------------------------------------------------------------------------------------------------------------
>             names
>     cluster2(s4App)                                        cluster1(myApp)
>     ----------------------------------------------------------------------------------------------------------------------------------
>
>
> *****************************************************************************************************************
>
>
>     Best Regards
>     Jihyoun Park
>
>
>
>
>     On Tue, Aug 28, 2012 at 4:33 PM, JiHyoun Park <april3@gmail.com
>     <mailto:april3@gmail.com>> wrote:
>
>         Dear Matthieu
>
>         Thank you for your comment.
>         I copied the s4r to the other machine and made the zk server
>         refer to the local instance....
>         This is my current temporary solution.
>         I will try your method.
>
>         Best Regards
>         Jihyoun Park
>
>
>
>         On Tue, Aug 28, 2012 at 3:57 PM, Matthieu Morel
>         <mmorel@apache.org <mailto:mmorel@apache.org>> wrote:
>
>             Hi,
>
>             the /tmp path is probably local to the hosts on which you
>             created the s4r file. Therefore, files under that path are
>             not accessible from the other host.
>
>             I suggest you copy/move the generated s4r to a shared
>             directory, and update the s4r location in Zookeeper. You may
>             also host as a static file in a web server.
>
>             See the -s4r option in the "s4 deploy" command.
>
>
>             Hope this helps,
>
>             Matthieu
>
>
>
>             On 8/28/12 4:16 AM, JiHyoun Park wrote:
>
>                 Hi,
>
>                 I tried to run 'myApp (the basic testing sample)' on 2
>                 machines (s4test4
>                 and s4test5).
>                 s4test5 is where the zkServer resides.
>                 I created 2 clusters (cluster1 with 2 nodes for app,
>                 cluster2 with 1
>                 node for adapter).
>                 Then I started 1 app node on s4test5 and another app
>                 node and an adapter
>                 node on s4test4.
>                 The remote nodes on s4test4 were successfully attached
>                 to the zkServer
>                 on s4test5 (pls refer to the s4 status attached at the
>                 bottom).
>                 However, when I deployed the app and the adapter on
>                 s4test5, I got this
>                 error message at the app node and the adater node in
>                 s4test4.
>
>                 org.apache.s4.deploy.DeploymentFailedException: Cannot
>                 retrieve file
>                 from uri [file:/tmp/1346061569272-0/myApp.s4r]
>
>                 Is it a zookeeper problem or what should I do?
>                 Below is the procedure that I followed to run the app
>                 using 2 machines.
>
>                 [@s4test5]
>                 ./s4 zkServer
>
>                 ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
>                 ./s4 newCluster -c=cluster2 -nbTasks=1 -flp=13000
>
>                 ./s4 node -c=cluster1
>
>                 [@s4test4]
>                 ./s4 node -c=cluster1 -zk=s4test5
>                 ./s4 node -c=cluster2 -p=s4.adapter.output.stream=names
>                   -zk=s4test5
>
>                 [@s4test5]
>                 ./s4 deploy -appName=myApp -c=cluster1 -b=`pwd`/build.gradle
>                 -a=hello.HelloApp
>                 ./s4 deploy -appName=s4App -c=cluster2 -b=`pwd`/build.gradle
>                 -a=hello.HelloInputAdapter
>
>                 echo "Bob" | nc localhost 15000
>
>
>                 [root@s4test4 apache-s4-0.5.0-incubating-src]# ./s4 node
>                 -c=cluster1
>                 -zk=s4test5
>                 18:00:52.655 [main] INFO  org.apache.s4.core.Main -
>                 Initializing S4 node
>                 with :
>                 - comm module class [org.apache.s4.comm.DefaultCommModule]
>                 - comm configuration file [default.s4.comm.properties
>                 from classpath]
>                 - core module class [org.apache.s4.core.DefaultCoreModule]
>                 - core configuration file[default.s4.core.properties
>                 from classpath]
>                 - extra modules: []
>                 - inline parameters: []
>                 18:00:52.664 [main] DEBUG org.apache.s4.core.Main -
>                 Adding named
>                 parameters for injection : [s4.cluster.zk_address=s4test5]
>                 18:00:52.953 [main] INFO  org.apache.s4.core.Main -
>                 Starting S4 node.
>                 This node will automatically download applications
>                 published for the
>                 cluster it belongs to
>                 18:00:53.025 [main] INFO
>                   o.a.s.comm.topology.AssignmentFromZK - New
>                 session:88215482878918664; state is : SyncConnected
>                 18:00:53.092 [main] INFO
>                   o.a.s.comm.topology.AssignmentFromZK -
>                 Successfully acquired task:Task-1 by s4test4
>                 0 [ZkClient-EventThread-13-s4test5] ERROR
>                 org.I0Itec.zkclient.ZkEventThread  - Error handling
>                 event ZkEvent[Data
>                 of /s4/clusters/cluster1/app/s4App changed sent to
>                 org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@10ba97c3]
>                 org.apache.s4.deploy.DeploymentFailedException: Cannot
>                 retrieve file
>                 from uri [file:/tmp/1346061569272-0/myApp.s4r]
>                       at
>                 org.apache.s4.deploy.FileSystemS4RFetcher.fetch(FileSystemS4RFetcher.java:38)
>                       at
>                 org.apache.s4.deploy.DistributedDeploymentManager.fetchS4App(DistributedDeploymentManager.java:151)
>                       at
>                 org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:119)
>                       at
>                 org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
>                       at
>                 org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
>                       at
>                 org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
>
>
>                 [root@s4test5 apache-s4-0.5.0-incubating-src]# ./s4 status
>
>                 App Status
>                 ----------------------------------------------------------------------------------------------------------------------------------
>                           Name
>                 Cluster                                                  URI
>                 ----------------------------------------------------------------------------------------------------------------------------------
>                          s4App              cluster2
>                 file:/tmp/1346061580311-0/s4App.s4r
>                          myApp             cluster1
>                   file:/tmp/1346061569272-0/myApp.s4r
>                 ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Cluster Status
>                 ----------------------------------------------------------------------------------------------------------------------------------
>
>                                               Active nodes
>                           Name                App           Tasks
>                 --------------------------------------------------------------------------------
>
>                 Number    Task
>                 id                         Host                         Port
>                 ----------------------------------------------------------------------------------------------------------------------------------
>                         cluster2             s4App           1         1
>                 Task-0                    s4test4                     13000
>                         cluster1             myApp          2         2
>                 Task-0                    s4test5                     12000
>
>                 Task-1                    s4test4                     12001
>                 ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Stream Status
>                 ----------------------------------------------------------------------------------------------------------------------------------
>                           Name
>                 Producers
>                   Consumers
>                 ----------------------------------------------------------------------------------------------------------------------------------
>                          names
>                 --
>                   cluster1(myApp)
>                 ----------------------------------------------------------------------------------------------------------------------------------
>
>
>                 Best Regards
>                 Jihyoun Park
>
>
>
>
>
>
>


Mime
View raw message