incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JiHyoun Park <apr...@gmail.com>
Subject Re: To deploy an app on multiple machines
Date Wed, 29 Aug 2012 02:15:01 GMT
Hi Matthieu

With the same source code (the Hello sample), I tried to test a 2-adapters
case using just one machine.
But when I deployed the adapter app (HelloInputAdapter), I got the
following error from one of adapters. (The other adapter was OK.)

************************************************************************************************************************************

> 09:43:14.310 [ZkClient-EventThread-13-localhost:2181] INFO
> org.apache.s4.core.Server - Loaded application from file
> /tmp/tmp3145808370992354402s4r
> 09:43:14.310 [ZkClient-EventThread-13-localhost:2181] INFO
> o.a.s.d.DistributedDeploymentManager - Successfully installed application
> s4App
> 09:43:14.311 [ZkClient-EventThread-13-localhost:2181] DEBUG
> o.a.s4.comm.topology.ClustersFromZK - Adding output stream [names] for app
> [-1] in cluster [cluster2]
> java.net.BindException: Address already in use
>     at java.net.PlainSocketImpl.socketBind(Native Method)
>     at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
>     at java.net.ServerSocket.bind(ServerSocket.java:376)
>     at java.net.ServerSocket.<init>(ServerSocket.java:237)
>     at java.net.ServerSocket.<init>(ServerSocket.java:128)
>     at hello.HelloInputAdapter$1.run(HelloInputAdapter.java:42)
>     at java.lang.Thread.run(Thread.java:722)

*******************************************************************************************************************

> Are there any instructions to run multiple adapters?
S4 status looks fine.

***********************************************************************************************************************

> [root@s4test4 myApp]# ./s4 status
> calling referenced s4 script : /root/apache-s4-0.5.0-incubating-src/s4
>
>
> App Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>         Name
> Cluster
> URI
>
> ----------------------------------------------------------------------------------------------------------------------------------
>        s4App              cluster2
> file:/tmp/1346204592699-0/s4App.s4r
>
>        myApp             cluster1
> file:/tmp/1346204586930-0/myApp.s4r
>
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Cluster Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Active nodes
>         Name                App           Tasks
> --------------------------------------------------------------------------------
>                                                    Number    Task
> id                         Host                         Port
>
> ----------------------------------------------------------------------------------------------------------------------------------
>       cluster2             s4App           2         2
> Task-0                    s4test4                     13000
>
> Task-1                    s4test4                     13001
>       cluster1             myApp           2         2
> Task-0                    s4test4                     12000
>
> Task-1                    s4test4                     12001
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Stream Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>         Name
> Producers
> Consumers
>
> ----------------------------------------------------------------------------------------------------------------------------------
>        names
> cluster2(s4App)
> cluster1(myApp)
>
> ----------------------------------------------------------------------------------------------------------------------------------
>

*****************************************************************************************************************

>
> Best Regards
> Jihyoun Park
>
>
>

On Tue, Aug 28, 2012 at 4:33 PM, JiHyoun Park <april3@gmail.com> wrote:
>
>> Dear Matthieu
>>
>> Thank you for your comment.
>> I copied the s4r to the other machine and made the zk server refer to the
>> local instance....
>> This is my current temporary solution.
>> I will try your method.
>>
>> Best Regards
>> Jihyoun Park
>>
>>
>>
>> On Tue, Aug 28, 2012 at 3:57 PM, Matthieu Morel <mmorel@apache.org>wrote:
>>
>>> Hi,
>>>
>>> the /tmp path is probably local to the hosts on which you created the
>>> s4r file. Therefore, files under that path are not accessible from the
>>> other host.
>>>
>>> I suggest you copy/move the generated s4r to a shared directory, and
>>> update the s4r location in Zookeeper. You may also host as a static file in
>>> a web server.
>>>
>>> See the -s4r option in the "s4 deploy" command.
>>>
>>>
>>> Hope this helps,
>>>
>>> Matthieu
>>>
>>>
>>>
>>> On 8/28/12 4:16 AM, JiHyoun Park wrote:
>>>
>>>> Hi,
>>>>
>>>> I tried to run 'myApp (the basic testing sample)' on 2 machines (s4test4
>>>> and s4test5).
>>>> s4test5 is where the zkServer resides.
>>>> I created 2 clusters (cluster1 with 2 nodes for app, cluster2 with 1
>>>> node for adapter).
>>>> Then I started 1 app node on s4test5 and another app node and an adapter
>>>> node on s4test4.
>>>> The remote nodes on s4test4 were successfully attached to the zkServer
>>>> on s4test5 (pls refer to the s4 status attached at the bottom).
>>>> However, when I deployed the app and the adapter on s4test5, I got this
>>>> error message at the app node and the adater node in s4test4.
>>>>
>>>> org.apache.s4.deploy.DeploymentFailedException: Cannot retrieve file
>>>> from uri [file:/tmp/1346061569272-0/myApp.s4r]
>>>>
>>>> Is it a zookeeper problem or what should I do?
>>>> Below is the procedure that I followed to run the app using 2 machines.
>>>>
>>>> [@s4test5]
>>>> ./s4 zkServer
>>>>
>>>> ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
>>>> ./s4 newCluster -c=cluster2 -nbTasks=1 -flp=13000
>>>>
>>>> ./s4 node -c=cluster1
>>>>
>>>> [@s4test4]
>>>> ./s4 node -c=cluster1 -zk=s4test5
>>>> ./s4 node -c=cluster2 -p=s4.adapter.output.stream=names  -zk=s4test5
>>>>
>>>> [@s4test5]
>>>> ./s4 deploy -appName=myApp -c=cluster1 -b=`pwd`/build.gradle
>>>> -a=hello.HelloApp
>>>> ./s4 deploy -appName=s4App -c=cluster2 -b=`pwd`/build.gradle
>>>> -a=hello.HelloInputAdapter
>>>>
>>>> echo "Bob" | nc localhost 15000
>>>>
>>>>
>>>> [root@s4test4 apache-s4-0.5.0-incubating-src]# ./s4 node -c=cluster1
>>>> -zk=s4test5
>>>> 18:00:52.655 [main] INFO  org.apache.s4.core.Main - Initializing S4 node
>>>> with :
>>>> - comm module class [org.apache.s4.comm.DefaultCommModule]
>>>> - comm configuration file [default.s4.comm.properties from classpath]
>>>> - core module class [org.apache.s4.core.DefaultCoreModule]
>>>> - core configuration file[default.s4.core.properties from classpath]
>>>> - extra modules: []
>>>> - inline parameters: []
>>>> 18:00:52.664 [main] DEBUG org.apache.s4.core.Main - Adding named
>>>> parameters for injection : [s4.cluster.zk_address=s4test5]
>>>> 18:00:52.953 [main] INFO  org.apache.s4.core.Main - Starting S4 node.
>>>> This node will automatically download applications published for the
>>>> cluster it belongs to
>>>> 18:00:53.025 [main] INFO  o.a.s.comm.topology.AssignmentFromZK - New
>>>> session:88215482878918664; state is : SyncConnected
>>>> 18:00:53.092 [main] INFO  o.a.s.comm.topology.AssignmentFromZK -
>>>> Successfully acquired task:Task-1 by s4test4
>>>> 0 [ZkClient-EventThread-13-s4test5] ERROR
>>>> org.I0Itec.zkclient.ZkEventThread  - Error handling event ZkEvent[Data
>>>> of /s4/clusters/cluster1/app/s4App changed sent to
>>>>
>>>> org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@10ba97c3
>>>> ]
>>>> org.apache.s4.deploy.DeploymentFailedException: Cannot retrieve file
>>>> from uri [file:/tmp/1346061569272-0/myApp.s4r]
>>>>      at
>>>>
>>>> org.apache.s4.deploy.FileSystemS4RFetcher.fetch(FileSystemS4RFetcher.java:38)
>>>>      at
>>>>
>>>> org.apache.s4.deploy.DistributedDeploymentManager.fetchS4App(DistributedDeploymentManager.java:151)
>>>>      at
>>>>
>>>> org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:119)
>>>>      at
>>>>
>>>> org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
>>>>      at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
>>>>      at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
>>>>
>>>>
>>>> [root@s4test5 apache-s4-0.5.0-incubating-src]# ./s4 status
>>>>
>>>> App Status
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>          Name
>>>> Cluster                                                  URI
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>         s4App              cluster2
>>>> file:/tmp/1346061580311-0/s4App.s4r
>>>>         myApp             cluster1
>>>>  file:/tmp/1346061569272-0/myApp.s4r
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Cluster Status
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>>              Active nodes
>>>>          Name                App           Tasks
>>>>
>>>> --------------------------------------------------------------------------------
>>>>                                                     Number    Task
>>>> id                         Host                         Port
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>        cluster2             s4App           1         1
>>>> Task-0                    s4test4                     13000
>>>>        cluster1             myApp          2         2
>>>> Task-0                    s4test5                     12000
>>>>
>>>> Task-1                    s4test4                     12001
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Stream Status
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>          Name
>>>> Producers                                              Consumers
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>         names
>>>> --                                              cluster1(myApp)
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> Best Regards
>>>> Jihyoun Park
>>>>
>>>
>>>
>>
>
>

Mime
View raw message