incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: S4R URL
Date Mon, 08 Oct 2012 09:30:26 GMT
Hi,

the log message that says it is fetching the app from a local temp 
directory is correct.
What each node does for loading S4 archives is a bit similar to what 
hadoop does when you deploy a new job. First, the node downloads the S4R 
from the remote URI. Then this is extracted to a local temporary 
directory. Then we define a new classloader that reads files from this 
temporary directory, so that S4R classes and dependencies can be 
dynamically loaded.
The log message is about the local temporary directory.

Regarding the logged error, this looks like a port conflict, most 
probably you have something else running on port 12000.

Hope this helps,

Matthieu





On 10/8/12 10:45 AM, Frank Zheng wrote:
> Hi All,
>
> I tried to run the Twitter Trending Example in a multi-server cluster on
> two machines.
> I followed the steps on the website except I create and deploy in two
> steps, instead of one, like this.
>
> ./s4 s4r
> -b=/usr/apache-s4-0.5.0-incubating-src/test-apps/twitter-counter/build.gradle
> twitter-counter
>
> ./s4 deploy
> -s4r=/usr/apache-s4-0.5.0-incubating-src/test-apps/twitter-counter/build/libs/twitter-counter.s4r
> -c=cluster1 -appName=twitter-counter -zk=testing.machine1:2181
>
>
> These two steps succeeded. And the status like this.
>
> App Status
> ----------------------------------------------------------------------------------------------------------------------------------
>          Name
> Cluster                                                  URI
> ----------------------------------------------------------------------------------------------------------------------------------
>    twitter-adapter         cluster2
> file:/usr/apache-s4-0.5.0-incubating-src/test-apps/twitter-adapter/build/libs/twitter-adapter.s4r
>    twitter-counter         cluster1
> file:/usr/apache-s4-0.5.0-incubating-src/test-apps/twitter-counter/build/libs/twitter-counter.s4r
> ----------------------------------------------------------------------------------------------------------------------------------
>
>
>
> Cluster Status
> ----------------------------------------------------------------------------------------------------------------------------------
>                                                                                     
Active nodes
>          Name                App           Tasks
> --------------------------------------------------------------------------------
>                                                     Number    Task
> id                         Host                         Port
> ----------------------------------------------------------------------------------------------------------------------------------
>        cluster2        twitter-adapter      1         0
>        cluster1        twitter-counter      2         0
> ----------------------------------------------------------------------------------------------------------------------------------
>
>
>
> Stream Status
> ----------------------------------------------------------------------------------------------------------------------------------
>          Name
> Producers                                              Consumers
> ----------------------------------------------------------------------------------------------------------------------------------
>       RawStatus
> --                                                     --
> ----------------------------------------------------------------------------------------------------------------------------------
>
>
> Then i set up three nodes, two for cluster1 and one cluster2.
> But they all loaded s4r file from "/tmp/XXXX", not the file directory I
> specified.
> The error messages are as follows.
>
> [root@testing apache-s4-0.5.0-incubating-src]# ./s4 node -c=cluster1
> -zk=testing.machine1:2181
> 16:36:26.655 [main] INFO  org.apache.s4.core.Main - Initializing S4 node
> with :
> - comm module class [org.apache.s4.comm.DefaultCommModule]
> - comm configuration file [default.s4.comm.properties from classpath]
> - core module class [org.apache.s4.core.DefaultCoreModule]
> - core configuration file[default.s4.core.properties from classpath]
> - extra modules: []
> - inline parameters: []
> 16:36:26.665 [main] DEBUG org.apache.s4.core.Main - Adding named
> parameters for injection : [s4.cluster.zk_address=testing.machine1:2181]
> 16:36:26.991 [main] INFO  org.apache.s4.core.Main - Starting S4 node.
> This node will automatically download applications published for the
> cluster it belongs to
> 16:36:32.087 [main] INFO  o.a.s.comm.topology.AssignmentFromZK - New
> session:88452960352141328; state is : SyncConnected
> 16:36:32.172 [main] INFO  o.a.s.comm.topology.AssignmentFromZK -
> Successfully acquired task:Task-0 by testing.machine1
> *16:36:32.200 [main] INFO  org.apache.s4.core.Server - Loading
> application [twitter-counter] from file [/tmp/tmp3230139471078202749s4r]
> 16:36:32.201 [main] WARN  o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir
> not specified, using temporary directory [/tmp/1349685392201-0] for
> unpacking S4R. You may want to specify a parent non-temporary directory.*
>
> 16:36:32.201 [main] INFO  o.a.s4.base.util.S4RLoaderFactory - Unzipping
> S4R archive in [/tmp/1349685392201-0]
> 16:36:32.295 [main] INFO  org.apache.s4.core.Server - App class name is:
> org.apache.s4.example.twitter.TwitterCounterApp
> 16:36:32.355 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Changing
> cluster topology to {
> nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=testing.machine1,taskId=Task-0}]}
> from null
> 16:36:32.387 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Adding
> topology change listener:org.apache.s4.comm.tcp.TCPEmitter@2e879860
> 16:36:32.433 [main] INFO  o.a.s4.comm.topology.ClustersFromZK - New
> session:88452960352141330
> 16:36:32.435 [main] INFO  o.a.s4.comm.topology.ClustersFromZK - Detected
> new stream [RawStatus]
> 16:36:32.468 [main] INFO  o.a.s4.comm.topology.ClustersFromZK - New
> session:88452960352141331
> 16:36:32.481 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Changing
> cluster topology to {
> nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=testing.machine1,taskId=Task-0}]}
> from null
> 16:36:32.489 [main] INFO  o.a.s4.comm.topology.ClusterFromZK - Changing
> cluster topology to {
> nbNodes=1,name=cluster2,mode=unicast,type=,nodes=[{partition=0,port=13000,machineName=testing.machine1,taskId=Task-0}]}
> from null
> 16:36:32.508 [main] ERROR org.apache.s4.core.Server - Could not load s4
> application form s4r file [{/tmp/tmp3230139471078202749s4r}]
> com.google.inject.ProvisionException: Guice provision errors:
>
> 1) Error injecting constructor,
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> 0.0.0.0/0.0.0.0:12000 <http://0.0.0.0/0.0.0.0:12000>
>    at org.apache.s4.comm.tcp.TCPListener.<init>(TCPListener.java:66)
>    while locating org.apache.s4.comm.tcp.TCPListener
>    while locating org.apache.s4.base.Listener
>      for parameter 0 at org.apache.s4.core.Receiver.<init>(Receiver.java:59)
>    at org.apache.s4.core.Receiver.class(Receiver.java:51)
>    while locating org.apache.s4.core.Receiver
>      for field at org.apache.s4.core.App.receiver(App.java:47)
>    while locating org.apache.s4.example.twitter.TwitterCounterApp
>
> 1 error
>      at
> com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944)
> ~[guice-3.0.jar:na]
>      at org.apache.s4.core.Server.loadApp(Server.java:120)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at
> org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at
> org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.core.Server.start(Server.java:87)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.core.Main.startNode(Main.java:150)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.core.Main.main(Main.java:75)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.6.0_22]
>      at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.6.0_22]
>      at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.6.0_22]
>      at java.lang.reflect.Method.invoke(Method.java:616) ~[na:1.6.0_22]
>      at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
> [s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.tools.Tools.main(Tools.java:94)
> [s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
> Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
> 0.0.0.0/0.0.0.0:12000 <http://0.0.0.0/0.0.0.0:12000>
>      at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:303)
> ~[netty-3.2.5.Final.jar:na]
>      at org.apache.s4.comm.tcp.TCPListener.<init>(TCPListener.java:92)
> ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
>      at
> org.apache.s4.comm.tcp.TCPListener$$FastClassByGuice$$82db237d.newInstance(<generated>)
> ~[guice-3.0.jar:0.5.0-incubating]
>      at
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> ~[guice-3.0.jar:na]
>      at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:75)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
> ~[guice-3.0.jar:na]
>      at
> com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
> ~[guice-3.0.jar:na]
>      ... 13 common frames omitted
> Caused by: java.net.BindException: Address already in use
>      at sun.nio.ch.Net.bind(Native Method) ~[na:1.6.0_22]
>      at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) ~[na:1.6.0_22]
>      at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> ~[na:1.6.0_22]
>      at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:148)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:100)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:74)
> ~[netty-3.2.5.Final.jar:na]
>      at org.jboss.netty.channel.Channels.bind(Channels.java:468)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:192)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:348)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:176)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.<init>(NioServerSocketChannel.java:85)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:142)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:90)
> ~[netty-3.2.5.Final.jar:na]
>      at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:282)
> ~[netty-3.2.5.Final.jar:na]
>      ... 36 common frames omitted
> 16:36:32.510 [main] ERROR o.a.s.d.DistributedDeploymentManager - Cannot
> deploy application
> org.apache.s4.deploy.DeploymentFailedException: Cannot deploy
> application [twitter-counter] from URI
> [file:/usr/apache-s4-0.5.0-incubating-src/test-apps/twitter-counter/build/libs/twitter-counter.s4r]
> : cannot start application
>      at
> org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
> ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at
> org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
> ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.core.Server.start(Server.java:87)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.core.Main.startNode(Main.java:150)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.core.Main.main(Main.java:75)
> [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.6.0_22]
>      at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.6.0_22]
>      at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.6.0_22]
>      at java.lang.reflect.Method.invoke(Method.java:616) ~[na:1.6.0_22]
>      at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
> [s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
>      at org.apache.s4.tools.Tools.main(Tools.java:94)
> [s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
> 16:36:51.598 [ZkClient-EventThread-25-testing.machine1:2181] INFO
> o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to {
> nbNodes=2,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=testing.machine1,taskId=Task-0},
> {partition=1,port=12001,machineName=testing.machine1,taskId=Task-1}]}
> from {
> nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=testing.machine1,taskId=Task-0}]}
> 16:36:51.601 [ZkClient-EventThread-19-testing.machine1:2181] INFO
> o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to {
> nbNodes=2,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=testing.machine1,taskId=Task-0},
> {partition=1,port=12001,machineName=testing.machine1,taskId=Task-1}]}
> from {
> nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=testing.machine1,taskId=Task-0}]}
>
>
>


Mime
View raw message