Yes they are.
I just noticed that the supervisor id is the same for both worker-node-1 and worker-node-2. Could that be because they were created from the same machine image?
David
From: Harsha [mailto:storm@harsha.io]
Sent: Friday, February 06, 2015 11:05 AM
To: user@storm.apache.org
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes
From you earlier nimbus config
storm.zookeeper.servers:
- "zookeeper-host-1"
- "zookeeper-host-2"
- "zookeeper-host-3"
and supervisor storm.yaml config looks like
storm.zookeeper.servers:
- "zookeeper-1"
- "zookeeper-2"
- "zookeeper-3"
Are they both pointing to same zk cluster?
-Harsha
On Fri, Feb 6, 2015, at 07:51 AM, David Shepherd wrote:
Sorry – that yaml is from the nimbus node that has no supervisor jobs for the supervisord process configured. I have different yaml om the worker nodes. The only difference in the yaml is the supervisor slot configuration. Also the worker machine only has the storm supervisor process configured to start in the supervisord configuration.
Nimbus node configuration
---------------------------------
storm.zookeeper.servers:
- "zookeeper-1"
- "zookeeper-2"
- "zookeeper-3"
storm.zookeeper.port: 2181
nimbus.host: "nimbus"
nimbus.thrift.port: 6627
storm.local.dir: "/var/opt/stormtmp"
java.library.path: "/usr/local/lib"
supervisor.slots.ports:
worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"
ui.childopts: "-Xmx512m
Worker node configuration
---------------------------------
storm.zookeeper.servers:
- "zookeeper-1"
- "zookeeper-2"
- "zookeeper-3"
storm.zookeeper.port: 2181
nimbus.host: "nimbus"
nimbus.thrift.port: 6627
storm.local.dir: "/var/opt/stormtmp"
java.library.path: "/usr/local/lib"
supervisor.slots.ports:
- 6700
- 6701
worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"
ui.childopts: "-Xmx512m"
On the nimbus node, the nimbus and ui process are running:
storm-nimbus RUNNING pid 2100, uptime 0:00:26
storm-ui RUNNING pid 2094, uptime 0:00:26
On the worker node 1 only the supervisor process is running
storm-supervisor RUNNING pid 2210, uptime 0:00:14
On the worker node 2 only the supervisor process is running
storm-supervisor RUNNING pid 1845, uptime 0:00:12
after starting the process on worker node 1 the supervisor log is:
2015-02-06T15:20:53.619+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-1.novalocal
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-1.novalocal
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-02-06T15:20:54.305+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
2015-02-06T15:20:54.330+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:20:54.416+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:20:54.418+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@457b9c87<mailto:watcher=org.apache.storm.curator.ConnectionState@457b9c87>
2015-02-06T15:20:54.439+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-2/10.0.0.6:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:20:54.445+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-2/10.0.0.6:2181, initiating session
2015-02-06T15:20:54.492+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-2/10.0.0.6:2181, sessionid = 0x14aefe7c7310013, negotiated timeout = 20000
2015-02-06T15:20:54.494+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:20:54.495+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-02-06T15:20:55.522+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-02-06T15:20:55.522+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x14aefe7c7310013 closed
2015-02-06T15:20:55.524+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:20:55.525+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:20:55.525+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d4e61f7<mailto:watcher=org.apache.storm.curator.ConnectionState@5d4e61f7>
2015-02-06T15:20:55.530+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:20:55.531+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181, initiating session
2015-02-06T15:20:55.540+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450011, negotiated timeout = 20000
2015-02-06T15:20:55.540+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:20:55.659+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host storm-worker-1.novalocal
And the storm ui looks like this:
[cid:image001.png@01D041FD.20625170]
after starting the process on worker node 2 the supervisor log is:
2015-02-06T15:39:16.901+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-2.novalocal
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-2.novalocal
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-02-06T15:39:17.580+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
2015-02-06T15:39:17.604+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:39:17.692+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:39:17.694+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@366186b4<mailto:watcher=org.apache.storm.curator.ConnectionState@366186b4>
2015-02-06T15:39:17.716+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:39:17.721+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181, initiating session
2015-02-06T15:39:17.745+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450012, negotiated timeout = 20000
2015-02-06T15:39:17.747+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:39:17.749+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-02-06T15:39:18.774+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-02-06T15:39:18.775+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x34aefe7a8450012 closed
2015-02-06T15:39:18.777+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:39:18.777+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:39:18.778+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@4501cefc<mailto:watcher=org.apache.storm.curator.ConnectionState@4501cefc>
2015-02-06T15:39:18.785+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:39:18.786+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session
2015-02-06T15:39:18.796+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001d, negotiated timeout = 20000
2015-02-06T15:39:18.796+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:39:18.863+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host storm-worker-2.novalocal
And the storm ui looks like this:
[cid:image002.png@01D041FD.20625170]
If I keep refreshing the UI, the host name toggles back and forth between storm-worker-1.novalocal and storm-worker-2.novalocal, the number of supervisors and slots stays and 1 and 2 respectively.
The supervisor.log is the only log file with any content the 2 worker nodes – the access and metric logs are empty.
The supervisor.log on the nimbus node is:
2015-02-06T15:09:48.692+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=nimbus
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-02-06T15:09:48.740+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=nimbus
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-02-06T15:09:49.883+0000 b.s.d.nimbus [INFO] Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" nil, "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
2015-02-06T15:09:49.909+0000 b.s.d.nimbus [INFO] Using default scheduler
2015-02-06T15:09:49.931+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:09:50.134+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:09:50.147+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d19fbae<mailto:watcher=org.apache.storm.curator.ConnectionState@5d19fbae>
2015-02-06T15:09:50.211+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:09:50.231+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session
2015-02-06T15:09:50.246+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001b, negotiated timeout = 20000
2015-02-06T15:09:50.248+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:09:50.250+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-02-06T15:09:51.270+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-02-06T15:09:51.271+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x24aefe7a813001b closed
2015-02-06T15:09:51.272+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:09:51.272+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:09:51.273+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@7cc1bdd7<mailto:watcher=org.apache.storm.curator.ConnectionState@7cc1bdd7>
2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session
2015-02-06T15:09:51.289+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001c, negotiated timeout = 20000
2015-02-06T15:09:51.289+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:09:51.324+0000 b.s.d.nimbus [INFO] Starting Nimbus server...
From: Harsha [mailto:storm@harsha.io]
Sent: Thursday, February 05, 2015 2:35 PM
To: user@storm.apache.org<mailto:user@storm.apache.org>
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes
David,
It looks like there might be some confusion between supervisor and workers.
"Zookeeper is running on separate VMs. 1 VM has the nimbus process, and the UI process, and no workers, and the other 2 machines each have a worker with 2 ports"
You have nimbus, UI and two supervisors and each with one worker slot. These workers starts running once you deploy a topology.
Number of workers per supervisor controlled by this config
supervisor.slots.ports:
- 6700
Since you specified only one port than it means each supervisor can only spawn one worker when a topology deployed. You can add more ports to increase the number of workers per supervisor.
"However when I go to the Storm UI, the cluster summary says there is only 1 supervisor with 2 slots, and the supervisor summary on has 1 row, but the host name switches between storm-worker-1 and storm-worker-2. It seems to be seeing both machines, but only 1 at a time."
I am assuming you are using same storm.yaml on all the hosts (i.e nimbus, supervisors etc) . In this case based on your config each supervisor should show it has one worker.
You should be able to see two supervisor listed under "Supervisor" section in storm UI. Do check the logs if your supervisors might be missing connection to zookeeper or crashing! .
Which version of storm you are using. It might help if you can attach screenshots for storm UI.
Thanks,
Harsha
On Thu, Feb 5, 2015, at 11:05 AM, David Shepherd wrote:
I have set up a Storm cluster on 3 vms running on openstack. Zookeeper is running on separate VMs. 1 VM has the nimbus process, and the UI process, and no workers, and the other 2 machines each have a worker with 2 ports. The workers were created by provisioning a VM from an image using the openstack tooling, so they are identical.
Everything starts up with no errors in any logs I can fin. However when I go to the Storm UI, the cluster summary says there is only 1 supervisor with 2 slots, and the supervisor summary on has 1 row, but the host name switches between storm-worker-1 and storm-worker-2. It seems to be seeing both machines, but only 1 at a time.
The storm.yaml is:
storm.zookeeper.servers:
- "zookeeper-host-1"
- "zookeeper-host-2"
- "zookeeper-host-3"
storm.zookeeper.port: 2181
nimbus.host: "nimbus"
nimbus.thrift.port: 6627
storm.local.dir: "/var/opt/stormtmp"
java.library.path: "/usr/local/lib"
supervisor.slots.ports:
- 6700
worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"
ui.childopts: "-Xmx512m"
I am totally stumped – can’t find anything on this behavior anywhere online or in any of the books I have. If anyone else has experienced this and can point me in the right direction it would be much appreciated.
Thanks,
David
Email had 2 attachments:
* image001.png
45k (image/png)
* image002.png
46k (image/png)
|