cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vedant (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-13972) Replication delays due to issues with inter node communication across multiple data centers, hints are piling up
Date Sat, 21 Oct 2017 19:10:00 GMT
Vedant created CASSANDRA-13972:
----------------------------------

             Summary: Replication delays due to issues with inter node communication across
multiple data centers, hints are piling up
                 Key: CASSANDRA-13972
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13972
             Project: Cassandra
          Issue Type: Bug
          Components: Configuration
         Environment: This is for Cassandra 2.1.13. At times there are replication delays
across multiple regions. Data is available (getting queried from command line) in 1 region
but not seen in other region(s).  This is not consistent. It is cluster spanning multiple
data centers with total > 30 nodes. Keyspace is configured to get replicated in all the
data centers. 

Hints are getting piled up in the source region. This happens especially for large data payload
(appro 1kb to few MB blobs).  Network  level congestion or saturation does not seem to be
an issue.  There is no memory/cpu pressure on individual nodes. 

I am attaching Cassandra.yaml, any pointers on what can be tuned are highly appreciated. Let
me know if you need any other info. 

We tried bumping up hinted_handoff_throttle_in_kb: 30720 and handoff tends to be slower max_hints_delivery_threads:
12 on one of the nodes to see if it speeds up hints delivery, there was some improvement but
not whole lot. 
            Reporter: Vedant
            Priority: Blocker






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message