cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prakash Chauhan <prakash.chau...@ericsson.com>
Subject RE: Slowness in C* cluster after implementing multiple network interface configuration.
Date Wed, 24 May 2017 07:12:50 GMT
Hi All,

We have a new observation.

Earlier for implementing multiple network interfaces, we were deleting cassandra-topologies.properties
in the last step (Steps are mentioned in mail trail).
The rationale was that because we are using altogether a new endpoint_snitch , we don't require
cassandra-topologies.properties file anymore.

Now we have observed that if we don't delete cassandra-topologies.properties, the slowness
is not there in the cluster (Even with multiple restarts)

Is there some relationship between GossipingPropertyFileSnitch and cassandra-topologies.properties
?

As per my knowledge,  cassandra-topologies.properties file is only used as a fallback while
doing snitch migration. If that's the case, why does Cassandra becomes slow with time ( and
after doing multiple restarts ) after deleting cassandra-topologies.properties ?




Regards,
Prakash Chauhan.

From: Cogumelos Maravilha [mailto:cogumelosmaravilha@sapo.pt]
Sent: Wednesday, May 24, 2017 12:15 AM
To: user@cassandra.apache.org
Subject: Re: Slowness in C* cluster after implementing multiple network interface configuration.


Hi,

I never used version 2.0.x but I think port 7000 isn't enough.

Try enable:

7000 inter-node

7001 SSL inter-node

9042 CQL

9160 Thrift is enable in that version



And

In Cassandra.yaml, add property "broadcast_address".  = local ipv4

In Cassandra.yaml, change "listen_address" to private IP. = local ipv4



As a starting point.



Cheers.

On 22-05-2017 12:36, Prakash Chauhan wrote:
Hi All ,

Need Help !!!

Setup Details:
Cassandra 2.0.14
Geo Red setup

*         DC1 - 3 nodes

*         DC2 - 3 nodes


We were trying to implement multiple network interfaces with Cassandra 2.0.14
After doing all the steps mentioned in DataStax doc http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/configuration/configMultiNetworks.html,
we observed that nodes were not able to see each other (checked using nodetool status).

To resolve this issue, we followed the comment<https://issues.apache.org/jira/browse/CASSANDRA-9748?focusedCommentId=14903515&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14903515>
mentioned in the JIRA : CASSANDRA-9748<https://issues.apache.org/jira/browse/CASSANDRA-9748>

Exact steps that we followed are :
--------------------------------

1.       Stop Cassandra

2.       Add rule to "iptables" to forward all packets on the public interface to the private
interface.


COMMAND: # iptables -t nat -A PREROUTING -p tcp -m tcp -d <destination> --dport 7000
-j DNAT --to-destination <to-destination>:7000



3.       In Cassandra.yaml, add property "broadcast_address".

4.       In Cassandra.yaml, change "listen_address" to private IP.

5.       Clear the data from directory "peers".

6.       Change Snitch to GossipingPropertyFileSnitch.

7.       Append following property to the file "/etc/cassandra/conf/cassandra-env.sh" to purge
gossip state.

JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"



8.       Start Cassandra

9.       After node has been started, remove following property from the file "/etc/cassandra/conf/cassandra-env.sh"
(previously added in step 7)

JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"

10.   Delete file "/etc/cassandra/conf/cassandra-topology.properties"


Now We have an observation that after multiple restarts of Cassandra on multiple nodes, slowness
is observed in the cluster.
The problem gets resolved when we revert the steps mentioned above.

Do u think there is any step that can cause the problem ?
We are suspecting Step 2(iptable rule) but not very sure about it.


Regards,
Prakash Chauhan.


Mime
View raw message