cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11540) The JVM should exit if jmx fails to bind
Date Thu, 05 May 2016 22:56:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273271#comment-15273271
] 

Michael Shuler edited comment on CASSANDRA-11540 at 5/5/16 10:56 PM:
---------------------------------------------------------------------

Just a quick note on behavior after this commit, as we have found in cstar_perf (trunk will
not start at all, currently).

If a user happens to set an environment variable {{JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=7199
..."}} startup fails. There's no other Cassandra running and nothing was listening to 7199
prior to startup.

I believe that commit 7b0c716 introduced for CASSANDRA-10091 does something a little wonky
with regards to JVM_OPTS. The previous commit, ad7e36b, does not throw an error when JVM_OPTS
are passed in the user env.

If this hard failure is working as intended, please let me know!

Repro on trunk HEAD:
{noformat}
(trunk)mshuler@mana:~/git/cassandra$ sudo netstat -atunp | grep 7199
(trunk)mshuler@mana:~/git/cassandra$
(trunk)mshuler@mana:~/git/cassandra$ export JVM_OPTS="-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=127.0.0.1"
(trunk)mshuler@mana:~/git/cassandra$ 
(trunk)mshuler@mana:~/git/cassandra$ cassandra -f
<...>
INFO  22:00:18 Not submitting build tasks for views in keyspace system as storage service
is not initialized
ERROR 22:00:18 Port already in use: 7199; nested exception is: 
        java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.8.0_92]
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) ~[na:1.8.0_92]
        at java.net.ServerSocket.bind(ServerSocket.java:375) ~[na:1.8.0_92]
        at java.net.ServerSocket.<init>(ServerSocket.java:237) ~[na:1.8.0_92]
        at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
~[na:1.8.0_92]
        at org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:21)
~[main/:na]
        at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) ~[na:1.8.0_92]
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:330) ~[na:1.8.0_92]
        at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:249) ~[na:1.8.0_92]
        at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) ~[na:1.8.0_92]
        at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) ~[na:1.8.0_92]
        at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:212) ~[na:1.8.0_92]
        at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:152) ~[na:1.8.0_92]
        at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:112) ~[na:1.8.0_92]
        at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239) ~[na:1.8.0_92]
        at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:94)
~[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:124)
[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:193) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568)
[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [main/:na]
(trunk)mshuler@mana:~/git/cassandra$
{noformat}


was (Author: mshuler):
Just a quick note on behavior after this commit, as we have found in cstar_perf (trunk will
not start at all, currently).

If a user happens to set and environment variable {{JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=7199
..."}} startup fails. There's no other Cassandra running and nothing was listening to 7199
prior to startup.

I believe that commit 7b0c716 introduced for CASSANDRA-10091 does something a little wonky
with regards to JVM_OPTS. The previous commit, ad7e36b, does not throw an error when JVM_OPTS
are passed in the user env.

If this hard failure is working as intended, please let me know!

Repro on trunk HEAD:
{noformat}
(trunk)mshuler@mana:~/git/cassandra$ sudo netstat -atunp | grep 7199
(trunk)mshuler@mana:~/git/cassandra$
(trunk)mshuler@mana:~/git/cassandra$ export JVM_OPTS="-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=127.0.0.1"
(trunk)mshuler@mana:~/git/cassandra$ 
(trunk)mshuler@mana:~/git/cassandra$ cassandra -f
<...>
INFO  22:00:18 Not submitting build tasks for views in keyspace system as storage service
is not initialized
ERROR 22:00:18 Port already in use: 7199; nested exception is: 
        java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.8.0_92]
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) ~[na:1.8.0_92]
        at java.net.ServerSocket.bind(ServerSocket.java:375) ~[na:1.8.0_92]
        at java.net.ServerSocket.<init>(ServerSocket.java:237) ~[na:1.8.0_92]
        at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
~[na:1.8.0_92]
        at org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:21)
~[main/:na]
        at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) ~[na:1.8.0_92]
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:330) ~[na:1.8.0_92]
        at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:249) ~[na:1.8.0_92]
        at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) ~[na:1.8.0_92]
        at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) ~[na:1.8.0_92]
        at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:212) ~[na:1.8.0_92]
        at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:152) ~[na:1.8.0_92]
        at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:112) ~[na:1.8.0_92]
        at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239) ~[na:1.8.0_92]
        at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:94)
~[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:124)
[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:193) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568)
[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [main/:na]
(trunk)mshuler@mana:~/git/cassandra$
{noformat}

> The JVM should exit if jmx fails to bind
> ----------------------------------------
>
>                 Key: CASSANDRA-11540
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11540
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Brandon Williams
>            Assignee: Alex Petrov
>              Labels: lhf
>             Fix For: 2.2.7, 3.7, 3.0.7
>
>
> If you are already running a cassandra instance, but for some reason try to start another
one, this happens:
> {noformat}
> INFO  20:57:09 JNA mlockall successful
> WARN  20:57:09 JMX is not enabled to receive remote connections. Please see cassandra-env.sh
for more info.
> ERROR 20:57:10 Error starting local jmx server:
> java.rmi.server.ExportException: Port already in use: 7199; nested exception is:
>         java.net.BindException: Address already in use
>         at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:340) ~[na:1.7.0_76]
>         at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:248) ~[na:1.7.0_76]
>         at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) ~[na:1.7.0_76]
>         at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) ~[na:1.7.0_76]
>         at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207) ~[na:1.7.0_76]
>         at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:122) ~[na:1.7.0_76]
>         at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:98) ~[na:1.7.0_76]
>         at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239) ~[na:1.7.0_76]
>         at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:100)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:222)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653)
[main/:na]
> Caused by: java.net.BindException: Address already in use
>         at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.7.0_76]
>         at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) ~[na:1.7.0_76]
>         at java.net.ServerSocket.bind(ServerSocket.java:376) ~[na:1.7.0_76]
>         at java.net.ServerSocket.<init>(ServerSocket.java:237) ~[na:1.7.0_76]
>         at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
~[na:1.7.0_76]
>         at org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:13)
~[main/:na]
>         at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) ~[na:1.7.0_76]
>         at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:329) ~[na:1.7.0_76]
>         ... 11 common frames omitted
> {noformat}
> However the startup continues, and ends up replaying commitlogs, which is probably not
a good thing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message