cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Issue with New Production Cluster
Date Fri, 29 Sep 2017 15:32:23 GMT
I don't know what logging is available driver side, I'd probably be writing
a shell script to ping all three servers to see if I can prove there's a
network problem outside of cassandra first.

On Fri, Sep 29, 2017 at 8:19 AM, Jonathan Baynes <
Jonathan.Baynes@tradeweb.com> wrote:

> Thank you Jeff, you are a credit to this community.. very helpful
>
>
>
> Last question Is there any logging I can turn on (especially with the
> cassandra drive 2.5.0) that would assist in gaining more of an insight as
> to what’s going on? If I’m getting any failed connections at the cassandra
> layer?
>
>
>
> Thanks
>
> J
>
>
>
> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com]
> *Sent:* 29 September 2017 16:07
>
> *To:* cassandra
> *Subject:* Re: Issue with New Production Cluster
>
>
>
> The failure detector is seeing updates every 2.1-2.5  seconds, which it
> will ignore because it's over the 2 second default failure detector
> interval.
>
>
>
> If there's no load, there's no reason it shouldn't be seeing it far more
> frequently.
>
>
>
> If you're not seeing any signs of GC pauses (like GCInspector log lines),
> then it seems like you've got a network interface flapping - maybe STP is
> triggering on a port, or a bad cable/interface, or something along those
> lines.
>
>
>
>
>
>
>
>
>
>
>
> On Fri, Sep 29, 2017 at 7:32 AM, Jonathan Baynes <
> Jonathan.Baynes@tradeweb.com> wrote:
>
> Hi Jeff,
>
>
>
> This is version 3.0.11. Being run on Oracle Red Hat Linux.
>
>
>
> If I retry immediately it fails, leave it for 20 minutes, like I have just
> now and retry it and it has worked. (?!?!)
>
> Ive checked all the logs (system and Debug ) and in the logs I have this:
>
>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:04,390 FailureDetector.java:456 -
> Ignoring interval time of 2102011879 <(210)%20201-1879> for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> INFO  [SharedPool-Worker-1] 2017-09-29 15:28:07,382 Message.java:615 -
> Unexpected exception during request; channel = [id: 0x963c334d, L:/
> 10.172.117.61:9042
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=>
> - R:/10.172.117.21:51853
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51853&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=NwQs5pv53MTu7RXO4tUrDSVZAtsEdXTsObmf9GWbNKE&e=>
> ]
>
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peerat io.netty.channel.unix.
> FileDescriptor.readAddress(...)(Unknown Source)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:07,389 FailureDetector.java:456 -
> Ignoring interval time of 2085147992 <(208)%20514-7992> for /10.172.117.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=>
>
> INFO  [SharedPool-Worker-2] 2017-09-29 15:28:07,400 Message.java:615 -
> Unexpected exception during request; channel = [id: 0xa7633234, L:/
> 10.172.117.61:9042
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=>
> ! R:/10.172.117.21:51818
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51818&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=Zd17m05nbPXla0hA2GWhNKHRRo-N_nFqOfrN2_c6meY&e=>
> ]
>
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peerat io.netty.channel.unix.
> FileDescriptor.readAddress(...)(Unknown Source)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
>
> INFO  [SharedPool-Worker-1] 2017-09-29 15:28:07,418 Message.java:615 -
> Unexpected exception during request; channel = [id: 0x8c685c2c, L:/
> 10.172.117.61:9042
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=>
> ! R:/10.172.117.21:51838
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51838&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=P-JajFY4WPK7GbEnPsK2RAoRmzPCztQnE8RT5atNnhY&e=>
> ]
>
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peerat io.netty.channel.unix.
> FileDescriptor.readAddress(...)(Unknown Source)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
>
> INFO  [HANDSHAKE-/10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>]
> 2017-09-29 15:28:10,140 OutboundTcpConnection.java:523 - Handshaking
> version with /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:12,288 FailureDetector.java:456 -
> Ignoring interval time of 2153356364 <(215)%20335-6364> for /10.172.117.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:15,289 FailureDetector.java:456 -
> Ignoring interval time of 2000150143 for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:16,305 FailureDetector.java:456 -
> Ignoring interval time of 2790916098 for /10.172.117.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:17,289 FailureDetector.java:456 -
> Ignoring interval time of 2000209617 for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:18,391 FailureDetector.java:456 -
> Ignoring interval time of 2085345347 <(208)%20534-5347> for /10.172.181.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:19,391 FailureDetector.java:456 -
> Ignoring interval time of 2102054326 <(210)%20205-4326> for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 FailureDetector.java:456 -
> Ignoring interval time of 2000754004 for /10.172.181.61
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 FailureDetector.java:456 -
> Ignoring interval time of 2000787648 for /10.172.181.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:23,391 FailureDetector.java:456 -
> Ignoring interval time of 2254752613 <(225)%20475-2613> for /10.172.117.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 FailureDetector.java:456 -
> Ignoring interval time of 2086206045 <(208)%20620-6045> for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 FailureDetector.java:456 -
> Ignoring interval time of 2086239333 <(208)%20623-9333> for /10.172.181.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:26,392 FailureDetector.java:456 -
> Ignoring interval time of 2000751862 for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:26,516 FailureDetector.java:456 -
> Ignoring interval time of 2124952635 <(212)%20495-2635> for /10.172.181.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:29,392 FailureDetector.java:456 -
> Ignoring interval time of 2254658974 <(225)%20465-8974> for /10.172.117.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:30,306 FailureDetector.java:456 -
> Ignoring interval time of 2000326847 for /10.172.181.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 FailureDetector.java:456 -
> Ignoring interval time of 2086445998 <(208)%20644-5998> for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 FailureDetector.java:456 -
> Ignoring interval time of 2086481727 <(208)%20648-1727> for /10.172.181.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 FailureDetector.java:456 -
> Ignoring interval time of 2363746015 <(236)%20374-6015> for /10.172.117.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 FailureDetector.java:456 -
> Ignoring interval time of 2151813348 for /10.172.117.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:37,393 FailureDetector.java:456 -
> Ignoring interval time of 2103432914 <(210)%20343-2914> for /10.172.181.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:40,394 FailureDetector.java:456 -
> Ignoring interval time of 2255155834 <(225)%20515-5834> for /10.172.117.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 FailureDetector.java:456 -
> Ignoring interval time of 2086966679 <(208)%20696-6679> for /10.172.117.62
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=>
>
> DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 FailureDetector.java:456 -
> Ignoring interval time of 2087037617 <(208)%20703-7617> for /10.172.181.63
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=>
>
> INFO  [HANDSHAKE-/10.172.181.61
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=>]
> 2017-09-29 15:28:47,132 OutboundTcpConnection.java:523 - Handshaking
> version with /10.172.181.61
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=>
>
>
>
>
>
> The Handshake @15:28 is about the same time as I just tried it and it
> working. So it has an inability to handshake with the nodes??
>
>
>
> Any suggestions would be great, I have to go to the Network team with
> this, but I’m not sure where to start…
>
>
>
> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com]
> *Sent:* 29 September 2017 14:54
> *To:* cassandra
> *Subject:* Re: Issue with New Production Cluster
>
>
>
> What version?
>
> If you retry immediately, does it reconnect?
>
> Anything in the logs?
>
>
>
> What you describe is atypical - timeouts on queries can (and will) happen
> occasionally under load, but timeout on connect is atypical. Any sign of
> networking issues/slowness/dns problems/etc?
>
>
>
>
>
> On Fri, Sep 29, 2017 at 6:43 AM, Jonathan Baynes <
> Jonathan.Baynes@tradeweb.com> wrote:
>
> Hi Community,
>
>
>
> I have a 6 node ring, covering 2 DC’s, the ring isn’t being used yet and
> we are just in the connectivity and testing phase. So the boxes are NOT
> under any load.
>
>
>
> I’ve gone to connect to CQLSH this afternoon and I’ve had this returned:
>
>
>
> cqlsh xx.xxx.xxx.xx -u cassandra -p cassandra
>
> Connection error: ('Unable to connect to any servers', { xx.xxx.xxx.xx’:
> OperationTimedOut('errors=Timed out creating connection (5 seconds),
> last_host=None',)})
>
>
>
> Status of the service is “Running”
>
> Nodetool status is UP for all nodes.
>
> Tested telnet on all ports and they are all up and connecting.
>
> Checked the PID is up for cassandra (it is)
>
>
>
> Any idea how I can debug the cause of this further?
>
>
>
>
>
> Thanks
>
> J
>
>
>
>
>
>
>
> *Jonathan Baynes*
>
> DBA
> Tradeweb Europe Limited
>
> Moor Place  •  1 Fore Street Avenue
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=>
>   •
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-250D-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-250D-2BEC2Y-2B9DT-2B-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-253Dhttps-2D3A-5F-5Fmaps.google.com-5F-2D3Fq-2D3D1-2D2BFore-2D2BStreet-2D2BAvenue-2D25C2-2D25A0-2D25C2-2D25A0-2D25E2-2D2580-2D25A2-2D25C2-2D25A0-2D25C2-2D25A0London-2D2BEC2Y-2D2B9DT-2D26entry-2D3Dgmail-2D26source-2D3Dg-2526d-253DDwMFaQ-2526c-253DsA0VaJZJFLZREu2pbPeqjXHJ-2DWd9NNzgHW3gpUOLSSk-2526r-253DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-2DdiQ8-2526m-253DynRqj6nQ2Rc0PoXhoOv-5F0HAaz8G8xuGO8wNbuHB-2D3pU-2526s-253D46cq-2D66-2DQUqnj-2DtaNMRFYe7YK3eNez6ogHJCv014X9w-2526e-253D-253E-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=a46k9sWzI3NKA2ZlZh8o78iGyheFPgXJFCmwuhYy-Qo&e=>
>   London EC2Y 9DT
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=>
> P +44 (0)20 77760988 <+44%2020%207776%200988>  •
> <https://maps.google.com/?q=1%0D+Fore+Street+Avenue%C2%A0%C2%A0%E2%80%A2+%3Chttps://urldefense.proofpoint.com/v2/url?u%3Dhttps-3A__maps.google.com_-3Fq-3D1-250D-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-250D-2BEC2Y-2B9DT-2B-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-253Dhttps-2D3A-5F-5Fmaps.google.com-5F-2D3Fq-2D3D1-2D2BFore-2D2BStreet-2D2BAvenue-2D25C2-2D25A0-2D25C2-2D25A0-2D25E2-2D2580-2D25A2-2D25C2-2D25A0-2D25C2-2D25A0London-2D2BEC2Y-2D2B9DT-2D26entry-2D3Dgmail-2D26source-2D3Dg-2526d-253DDwMFaQ-2526c-253DsA0VaJZJFLZREu2pbPeqjXHJ-2DWd9NNzgHW3gpUOLSSk-2526r-253DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-2DdiQ8-2526m-253DynRqj6nQ2Rc0PoXhoOv-5F0HAaz8G8xuGO8wNbuHB-2D3pU-2526s-253D46cq-2D66-2DQUqnj-2DtaNMRFYe7YK3eNez6ogHJCv014X9w-2526e-253D-253E-26entry-3Dgmail-26source-3Dg%26d%3DDwMFaQ%26c%3DsA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk%26r%3DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8%26m%3DLHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA%26s%3Da46k9sWzI3NKA2ZlZh8o78iGyheFPgXJFCmwuhYy-Qo%26e%3D%3E%C2%A0%C2%A0London%0D+EC2Y+9DT+%3Chttps://urldefense.proofpoint.com/v2/url?u%3Dhttps-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg%26d%3DDwMFaQ%26c%3DsA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk%26r%3DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8%26m%3DynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU%26s%3D46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w%26e%3D%3E&entry=gmail&source=g>
 F
> +44 (0)20 7776 3201 <+44%2020%207776%203201>  •  M +44 (0)xxxx xxxxxx
>
> Jonathan.Baynes@tradeweb.com
>
>
>
> [image: cid:image001.jpg@01CD26AD.4165F110] <http://www.tradeweb.com/>
> follow us:  *[image: cid:image002.jpg@01CD26AD.4165F110]*
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=saD3ZbF7LhTn3IHCHWSzhGp8Ns6SIRO0s0KXCnEbqLU&e=>
> [image: cid:image003.jpg@01CD26AD.4165F110]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=Duw_ud1x7HIX3h-0UEtKorCrlKn8C5tUoH6tkrvXDy0&e=>
>
> —————————————————————————————————————————
>
> A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for
> electronic fixed income, derivatives and ETF trading
>
>
>
> ________________________________________________________________________
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy it. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden. Tradeweb reserves the right to monitor all e-mail
> communications through its networks. If you do not wish to receive
> marketing emails about our products / services, please let us know by
> contacting us, either by email at contactus@tradeweb.com or by writing to
> us at the registered office of Tradeweb in the UK, which is: Tradeweb
> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
> 9DT
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=VSn9NuhaGsP_VCpKoBZFV2hz_46iJ3VhVk5rYBKHeN8&e=>.
> To see our privacy policy, visit our website @ www.tradeweb.com.
>
>
>
> ________________________________________________________________________
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy it. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden. Tradeweb reserves the right to monitor all e-mail
> communications through its networks. If you do not wish to receive
> marketing emails about our products / services, please let us know by
> contacting us, either by email at contactus@tradeweb.com or by writing to
> us at the registered office of Tradeweb in the UK, which is: Tradeweb
> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
> 9DT
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=m5TdlyiKgTyluGY2ssC5qtvdFJw9hAnGpfbDAKEHzxA&e=>.
> To see our privacy policy, visit our website @ www.tradeweb.com.
>
>
>
> ________________________________________________________________________
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy it. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden. Tradeweb reserves the right to monitor all e-mail
> communications through its networks. If you do not wish to receive
> marketing emails about our products / services, please let us know by
> contacting us, either by email at contactus@tradeweb.com or by writing to
> us at the registered office of Tradeweb in the UK, which is: Tradeweb
> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
> 9DT
> <https://maps.google.com/?q=number+3912826),+1+Fore+Street+Avenue+London+EC2Y+9DT&entry=gmail&source=g>.
> To see our privacy policy, visit our website @ www.tradeweb.com.
>

Mime
View raw message