storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Shepherd <...@nexjsystems.com>
Subject RE: Storm cluster only sees 1 worker node at a time - but swtiches between nodes
Date Fri, 06 Feb 2015 15:51:51 GMT
Sorry – that yaml is from the nimbus node that has no supervisor jobs for the supervisord
process configured.  I have different yaml om the worker nodes.  The only difference in the
yaml is the supervisor slot configuration.  Also the worker machine only has the storm supervisor
process configured to start in the supervisord configuration.

Nimbus node configuration
---------------------------------
storm.zookeeper.servers:
     - "zookeeper-1"
     - "zookeeper-2"
     - "zookeeper-3"
storm.zookeeper.port: 2181

nimbus.host: "nimbus"
nimbus.thrift.port: 6627
storm.local.dir: "/var/opt/stormtmp"
java.library.path: "/usr/local/lib"

supervisor.slots.ports:

worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"
ui.childopts: "-Xmx512m

Worker node configuration
---------------------------------
storm.zookeeper.servers:
     - "zookeeper-1"
     - "zookeeper-2"
     - "zookeeper-3"
storm.zookeeper.port: 2181

nimbus.host: "nimbus"
nimbus.thrift.port: 6627
storm.local.dir: "/var/opt/stormtmp"
java.library.path: "/usr/local/lib"

supervisor.slots.ports:
     - 6700
     - 6701

worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"
ui.childopts: "-Xmx512m"

On the nimbus node, the nimbus and ui process are running:
storm-nimbus                     RUNNING    pid 2100, uptime 0:00:26
storm-ui                         RUNNING    pid 2094, uptime 0:00:26

On the worker node 1 only the supervisor process is running
storm-supervisor                 RUNNING    pid 2210, uptime 0:00:14

On the worker node 2 only the supervisor process is running
storm-supervisor                 RUNNING    pid 1845, uptime 0:00:12

after starting the process on  worker node 1 the supervisor log is:
2015-02-06T15:20:53.619+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965,
built on 02/20/2014 09:09 GMT
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-1.novalocal
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle
Corporation
2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965,
built on 02/20/2014 09:09 GMT
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-1.novalocal
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle
Corporation
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-02-06T15:20:54.305+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path"
"/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs"
60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval"
5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads"
1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
"topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms"
10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib",
"topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size"
5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true,
"nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads"
64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate",
"topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms"
100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis"
30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers"
["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional",
"topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts"
nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs"
5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs"
3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs"
3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy"
"backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending"
nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1,
"nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
[6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs"
120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs"
10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate"
0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer",
"topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
"backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout"
15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count"
1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs"
60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport"
"backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms"
1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts"
"-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size"
262144, "topology.classpath" nil}
2015-02-06T15:20:54.330+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:20:54.416+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:20:54.418+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@457b9c87
2015-02-06T15:20:54.439+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server
zookeeper-2/10.0.0.6:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:20:54.445+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-2/10.0.0.6:2181,
initiating session
2015-02-06T15:20:54.492+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server
zookeeper-2/10.0.0.6:2181, sessionid = 0x14aefe7c7310013, negotiated timeout = 20000
2015-02-06T15:20:54.494+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:20:54.495+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-02-06T15:20:55.522+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-02-06T15:20:55.522+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x14aefe7c7310013 closed
2015-02-06T15:20:55.524+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:20:55.525+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:20:55.525+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm
sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d4e61f7
2015-02-06T15:20:55.530+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server
zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:20:55.531+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181,
initiating session
2015-02-06T15:20:55.540+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server
zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450011, negotiated timeout = 20000
2015-02-06T15:20:55.540+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:20:55.659+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f
at host storm-worker-1.novalocal

And the storm ui looks like this:
[cid:image001.png@01D041F9.73A268B0]
after starting the process on  worker node 2 the supervisor log is:
2015-02-06T15:39:16.901+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965,
built on 02/20/2014 09:09 GMT
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-2.novalocal
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle
Corporation
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965,
built on 02/20/2014 09:09 GMT
2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-2.novalocal
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle
Corporation
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-02-06T15:39:17.580+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path"
"/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs"
60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval"
5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads"
1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
"topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms"
10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib",
"topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size"
5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true,
"nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads"
64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate",
"topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms"
100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis"
30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers"
["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional",
"topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts"
nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs"
5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs"
3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs"
3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy"
"backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending"
nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1,
"nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
[6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs"
120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs"
10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate"
0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer",
"topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
"backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout"
15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count"
1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs"
60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport"
"backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms"
1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts"
"-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size"
262144, "topology.classpath" nil}
2015-02-06T15:39:17.604+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:39:17.692+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:39:17.694+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@366186b4
2015-02-06T15:39:17.716+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server
zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:39:17.721+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181,
initiating session
2015-02-06T15:39:17.745+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server
zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450012, negotiated timeout = 20000
2015-02-06T15:39:17.747+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:39:17.749+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-02-06T15:39:18.774+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-02-06T15:39:18.775+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x34aefe7a8450012 closed
2015-02-06T15:39:18.777+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:39:18.777+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:39:18.778+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm
sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@4501cefc
2015-02-06T15:39:18.785+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server
zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:39:18.786+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181,
initiating session
2015-02-06T15:39:18.796+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server
zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001d, negotiated timeout = 20000
2015-02-06T15:39:18.796+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:39:18.863+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f
at host storm-worker-2.novalocal

And the storm ui looks like this:

[cid:image002.png@01D041F9.F009F170]

If I keep refreshing the UI, the host name toggles back and forth between storm-worker-1.novalocal
and storm-worker-2.novalocal, the number of supervisors and slots stays and 1 and 2 respectively.

The supervisor.log is the only log file with any content the 2 worker nodes – the access
and metric logs are empty.

The supervisor.log on the nimbus node is:
2015-02-06T15:09:48.692+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965,
built on 02/20/2014 09:09 GMT
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=nimbus
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle
Corporation
2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-02-06T15:09:48.740+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965,
built on 02/20/2014 09:09 GMT
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=nimbus
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle
Corporation
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-02-06T15:09:49.883+0000 b.s.d.nimbus [INFO] Starting Nimbus with conf {"dev.zookeeper.path"
"/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs"
60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval"
5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads"
1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
"topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms"
10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib",
"topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size"
5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true,
"nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads"
64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate",
"topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms"
100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis"
30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers"
["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional",
"topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts"
nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs"
5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs"
3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs"
3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy"
"backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending"
nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1,
"nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
nil, "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs"
60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1,
"supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate"
0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer",
"topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
"backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout"
15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count"
1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs"
60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport"
"backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms"
1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts"
"-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size"
262144, "topology.classpath" nil}
2015-02-06T15:09:49.909+0000 b.s.d.nimbus [INFO] Using default scheduler
2015-02-06T15:09:49.931+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:09:50.134+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:09:50.147+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d19fbae
2015-02-06T15:09:50.211+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server
zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:09:50.231+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181,
initiating session
2015-02-06T15:09:50.246+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server
zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001b, negotiated timeout = 20000
2015-02-06T15:09:50.248+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:09:50.250+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-02-06T15:09:51.270+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-02-06T15:09:51.271+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x24aefe7a813001b closed
2015-02-06T15:09:51.272+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-02-06T15:09:51.272+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-02-06T15:09:51.273+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm
sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@7cc1bdd7
2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server
zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181,
initiating session
2015-02-06T15:09:51.289+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server
zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001c, negotiated timeout = 20000
2015-02-06T15:09:51.289+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-02-06T15:09:51.324+0000 b.s.d.nimbus [INFO] Starting Nimbus server...


From: Harsha [mailto:storm@harsha.io]
Sent: Thursday, February 05, 2015 2:35 PM
To: user@storm.apache.org
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes

David,
         It looks like there might be some confusion between supervisor and workers.

"Zookeeper is running on separate VMs. 1 VM has the nimbus process, and the UI process, and
no workers, and the other 2 machines each have a worker with 2 ports"
You have nimbus, UI and two supervisors and each with one worker slot. These workers starts
running once you deploy a topology.
Number of workers per supervisor controlled by this config
supervisor.slots.ports:
  - 6700
Since you specified only one port than it means each supervisor can only spawn one worker
when a topology deployed. You can add more ports to increase the number of workers per supervisor.

"However when I go to the Storm UI, the cluster summary says there is only 1 supervisor with
2 slots,  and the supervisor summary on has 1 row, but the host name switches between storm-worker-1
and storm-worker-2. It seems to be seeing both machines, but only 1 at a time."

I am assuming you are using same storm.yaml on all the hosts (i.e nimbus, supervisors etc)
. In this case based on your config each supervisor should show it has one worker.
You should be able to see two supervisor listed under "Supervisor" section in storm UI.  Do
check the logs if your supervisors might be missing connection to zookeeper or crashing! .

Which version of storm you are using. It might help if you can attach screenshots for storm
UI.
Thanks,
Harsha



On Thu, Feb 5, 2015, at 11:05 AM, David Shepherd wrote:

I have set up a Storm cluster on 3 vms running on openstack.  Zookeeper is running on separate
VMs. 1 VM has the nimbus process, and the UI process, and no workers, and the other 2 machines
each have a worker with 2 ports.  The workers were created by provisioning a VM from an image
using the openstack tooling, so they are identical.



Everything starts up with no errors in any logs I can fin.  However when I go to the Storm
UI, the cluster summary says there is only 1 supervisor with 2 slots,  and the supervisor
summary on has 1 row, but the host name switches between storm-worker-1 and storm-worker-2.
It seems to be seeing both machines, but only 1 at a time.



The storm.yaml is:



storm.zookeeper.servers:

    - "zookeeper-host-1"

    - "zookeeper-host-2"

    - "zookeeper-host-3"

storm.zookeeper.port: 2181



nimbus.host: "nimbus"

nimbus.thrift.port: 6627

storm.local.dir: "/var/opt/stormtmp"

java.library.path: "/usr/local/lib"



supervisor.slots.ports:

     - 6700

worker.childopts: "-Xmx768m"

nimbus.childopts: "-Xmx512m"

supervisor.childopts: "-Xmx256m"

ui.childopts: "-Xmx512m"



I am totally stumped – can’t find anything on this behavior anywhere online or in any
of the books I have.  If anyone else has experienced this and can point me in the right direction
it would be much appreciated.



Thanks,



David



Mime
View raw message