storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonas Krauss <jkraus...@gmail.com>
Subject Re: Various configuration issues setting up a new cluster
Date Wed, 25 Nov 2020 11:03:50 GMT
After doing more testing and finding the nimbus logfile in the storm dir, I
was able to narrow down the problem to a bug in Zookeeper 3.4, which is
somehow still used by storm internally, and Java 14:

https://issues.apache.org/jira/browse/ZOOKEEPER-3779

Others seem to have the same problem:

https://stackoverflow.com/questions/63804162/apache-storm-local-cluster-unable-to-canonicalize-address

It would be helpful if anybody knew how to overcome this, either by using a
higher zookeeper version in the shaded deps or by setting some config
options so it goes away.

If someone from the storm devs thinks it's worth raising an issue I am glad
to contribute, if I can.

Thanks

Jonas


Am Di., 24. Nov. 2020 um 17:34 Uhr schrieb Jonas Krauss <jkrauss82@gmail.com
>:

> Hi all,
>
> we have recently tried to update zookeeper from 3.4.13 to 3.6.2 on our
> production storm cluster without success. Therefore, we are now tinkering
> with a test cluster, including just two nodes.
>
> The two nodes have the following setup:
>
> *OS* Ubuntu 20.04.1 LTS
> *Zookeeper *3.6.2
> *Storm *2.2.0
> *Java *OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
> *Python* 2.7.18
> *Python2* 2.7.18
> *Python3* 3.8.5
>
>
> We are following this guide *to the letter*:
> http://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html
>
> First problem here: when following the guide and setting the mandatory
> parameters, you will end up with a configuration where the UI does not work.
> This is because according to the defaults.yaml, *both ui.port and
> storm.exhibitor.port are set to 8080*. This is where I get my information
> in regards to the default values from:
> https://github.com/apache/storm/blob/master/conf/defaults.yaml
> Left unconfigured, the UI will not start because it cannot bind to port
> 8080. Instead, you will see the 404 page from jetty.
> It took me quite a while to figure this, so I would be glad the person in
> charge could amend the docs to reflect the necessity to set one of the two
> to something different.
>
> Next problem, which I am not able to solve, but from searching the web I
> understand many storm users are facing similar problems, is the dreaded *NimbusLeaderNotFoundException:
> Could not find leader nimbus from seed hosts*
> Some examples:
>
>
> https://stackoverflow.com/questions/36742451/apache-storm-could-not-find-leader-nimbus-from-seed-hosts
>
> https://stackoverflow.com/questions/58155057/how-to-fix-could-not-find-leader-nimbus-from-seed-hosts-192-168-23-165
>
> https://stackoverflow.com/questions/37480155/nimbusleadernotfoundexception-in-apache-storm
>
>
> I have double checked the ports are accessible on both nodes and that
> everything else is running as expected.
>
> Looking at the outputs from nimbus and supervisor, there is no error
> message (stderr is empty) and there is no useful logging in stdout, only
> the java command which starts the two processes. Upon accessing the UI in
> the browser, the process will load for a while, then, both nimbus and
> supervisor will crash/be killed/? and the error message appears. I have run
> the processes in supervision with supervisord and also have tried to start
> them via bash, to get more meaningful logging, no luck here.
>
> It is almost impossible to debug without more meaningful logging. My best
> guess is it has to do with that we
>
> Maybe I am missing something in the host configurations, but I am at a
> loss here so I would appreciate any pointers to where to look next.
>
> Attaching storm.yaml and hosts configurations below.
>
> Regards
>
> Jonas
>
> *storm.yaml*
>
> storm.zookeeper.servers:
>    - "stormtest1.example.com"
>    - "stormtest2.example.com"
>
> nimbus.seeds: ["stormtest1.example.com", "stormtest2.example.com"]
>
> storm.local.dir: "/storm"
> storm.zookeeper.root: "/zookeeper"
> supervisor.slots.ports:
>    - 6700
>    - 6701
>    - 6702
>    - 6703
>
> ui.port: 8888
>
>
> */etc/hosts (node 1)*
>
> 127.0.1.1 storm-1
> 127.0.0.1 localhost
> # Test Storm
> xxx.xxx.xxx.xxx    stormtest1.example.com        storm-test-1
> xxx.xxx.xxx.xxx    stormtest2.example.com        storm-test-2
>
>
>
> */etc/hosts (node 2)*
>
> 127.0.1.1 storm-2
> 127.0.0.1 localhost
> # Test Storm
> xxx.xxx.xxx.xxx    stormtest1.example.com        storm-test-1
> xxx.xxx.xxx.xxx    stormtest2.example.com        storm-test-2
>
>
>

Mime
View raw message