storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonas Krauss <jkraus...@gmail.com>
Subject Various configuration issues setting up a new cluster
Date Tue, 24 Nov 2020 16:34:28 GMT
Hi all,

we have recently tried to update zookeeper from 3.4.13 to 3.6.2 on our
production storm cluster without success. Therefore, we are now tinkering
with a test cluster, including just two nodes.

The two nodes have the following setup:

*OS* Ubuntu 20.04.1 LTS
*Zookeeper *3.6.2
*Storm *2.2.0
*Java *OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
*Python* 2.7.18
*Python2* 2.7.18
*Python3* 3.8.5


We are following this guide *to the letter*:
http://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html

First problem here: when following the guide and setting the mandatory
parameters, you will end up with a configuration where the UI does not work.
This is because according to the defaults.yaml, *both ui.port and
storm.exhibitor.port are set to 8080*. This is where I get my information
in regards to the default values from:
https://github.com/apache/storm/blob/master/conf/defaults.yaml
Left unconfigured, the UI will not start because it cannot bind to port
8080. Instead, you will see the 404 page from jetty.
It took me quite a while to figure this, so I would be glad the person in
charge could amend the docs to reflect the necessity to set one of the two
to something different.

Next problem, which I am not able to solve, but from searching the web I
understand many storm users are facing similar problems, is the
dreaded *NimbusLeaderNotFoundException:
Could not find leader nimbus from seed hosts*
Some examples:

https://stackoverflow.com/questions/36742451/apache-storm-could-not-find-leader-nimbus-from-seed-hosts
https://stackoverflow.com/questions/58155057/how-to-fix-could-not-find-leader-nimbus-from-seed-hosts-192-168-23-165
https://stackoverflow.com/questions/37480155/nimbusleadernotfoundexception-in-apache-storm


I have double checked the ports are accessible on both nodes and that
everything else is running as expected.

Looking at the outputs from nimbus and supervisor, there is no error
message (stderr is empty) and there is no useful logging in stdout, only
the java command which starts the two processes. Upon accessing the UI in
the browser, the process will load for a while, then, both nimbus and
supervisor will crash/be killed/? and the error message appears. I have run
the processes in supervision with supervisord and also have tried to start
them via bash, to get more meaningful logging, no luck here.

It is almost impossible to debug without more meaningful logging. My best
guess is it has to do with that we

Maybe I am missing something in the host configurations, but I am at a loss
here so I would appreciate any pointers to where to look next.

Attaching storm.yaml and hosts configurations below.

Regards

Jonas

*storm.yaml*

storm.zookeeper.servers:
   - "stormtest1.example.com"
   - "stormtest2.example.com"

nimbus.seeds: ["stormtest1.example.com", "stormtest2.example.com"]

storm.local.dir: "/storm"
storm.zookeeper.root: "/zookeeper"
supervisor.slots.ports:
   - 6700
   - 6701
   - 6702
   - 6703

ui.port: 8888


*/etc/hosts (node 1)*

127.0.1.1 storm-1
127.0.0.1 localhost
# Test Storm
xxx.xxx.xxx.xxx    stormtest1.example.com        storm-test-1
xxx.xxx.xxx.xxx    stormtest2.example.com        storm-test-2



*/etc/hosts (node 2)*

127.0.1.1 storm-2
127.0.0.1 localhost
# Test Storm
xxx.xxx.xxx.xxx    stormtest1.example.com        storm-test-1
xxx.xxx.xxx.xxx    stormtest2.example.com        storm-test-2

Mime
View raw message