cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan <cne...@yahoo.com>
Subject Re: When are hints written?
Date Fri, 22 Apr 2016 02:33:37 GMT
HI Bo; 

you raised 2 questions: 
20% system utilization
Hints 

20% system utilization:  For a node or a cluster to have 20% utilization is Normal during
peak write operation.  
Hints:       hints are written when a node is unreachable;    C* 3.0  has a complete over
haul in the way hints have been implemented. 

Recommend reading up this blog article: 
http://www.datastax.com/dev/blog/whats-coming-to-cassandra-in-3-0-improved-hint-storage-and-delivery

hope this helps
Jan/


--------------------------------------------
On Thu, 4/21/16, Jens Rantil <jens.rantil@tink.se> wrote:

 Subject: Re: When are hints written?
 To: user@cassandra.apache.org
 Date: Thursday, April 21, 2016, 8:57 AM
 
 Hi again
 Bo,
 I assume this is the piece of
 documentation you are referring to? http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_about_hh_c.html?scroll=concept_ds_ifg_jqx_zj__performance
 
 > If a
 replica node is overloaded or unavailable, and the failure
 detector has not yet marked it down, then expect most or all
 writes to that node to fail after the timeout triggered by
 write_request_timeout_in_ms,
 which defaults to 10 seconds. During that time, Cassandra
 writes the hint when the timeout is reached.
 I'm not an expert on this, but
 the way I've seen is that hints are written stored as
 soon as there is _any_ issues writing a mutation
 (insert/update/delete) to a node. By "issue", that
 essentially means that a node hasn't acknowledged back
 to the coordinator that the write succeeded within
 write_request_timeout_in_ms. This includes TCP/socket
 timeouts, connection issues or that the node is down. The
 hints are stored for a maximum timespan defaulting to 3
 hours.
 
 Cheers,
 Jens
 On Thu, Apr
 21, 2016 at 8:06 AM Bo Finnerup Madsen <bo.gundersen@gmail.com>
 wrote:
 Hi Jens,
 Thank you
 for the tip!ALL would definitely cure our hints
 issue, but as you note, it is not optimal as we are unable
 to take down nodes without clients failing.
 I am most probably overlooking
 something in the documentation, but I cannot see any
 description of when hints are written other than when a node
 is marked as being down. And since none of our nodes have
 been marked as being down (at least according to the logs),
 I suspect that there is some timeout that governs when hints
 are written?
 Regarding
 your other post: Yes, 3.0.3 is pretty new. But we are new to
 this cassandra game, and our schema-fu is not strong enough
 for us to create a schema without using materialized views
 :)
 
 ons. 20. apr. 2016 kl. 17.09 skrev Jens Rantil
 <jens.rantil@tink.se>:
 Hi Bo,
 > In our case, I would like for the
 cluster to wait for the write to be persisted on the
 relevant nodes before returning an ok to the client. But I don't know which
 knobs to turn to accomplish this? or if it is even possible
 :)
 This is what write consistency
 option is for. Have a look at https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html.
 Note, however that if you use ALL, your clients will fail
 (throw exception, depending on language) as soon as a single
 partition can't be written. This means you can't do
 online maintenance of a Cassandra node (such as upgrading it
 etc.) without experiencing write issues.
 Cheers,Jens
 On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen
 <bo.gundersen@gmail.com>
 wrote:
 Hi,
 We have a
 small 5 node cluster of m4.xlarge clients that receives
 writes from ~20 clients. The clients will write as fast as
 they can, and the whole process is limited by the write
 performance of the cassandra cluster.After we have tweaked our schema to
 avoid large partitions, the load is going ok and we
 don't see any warnings or errors in the cassandra logs.
 But we do see quite a lot of hint handoff activity. During
 the load, the cassandra nodes are quite loaded, with linux
 reporting a load as high as 20.
 I have read the available
 documentation on how hints works, and to my understanding
 hints should only be written if a node is down. But as far
 as I can see, none of the nodes are marked as down during
 the load. So I suspect I am missing something
 :)We have configured the servers
 with write_request_timeout_in_ms: 120000 and the clients
 with a timeout of 130000, but still get hints
 stored.
 In our case, I
 would like for the cluster to wait for the write to be
 persisted on the relevant nodes before returning an ok to
 the client. But I don't know which knobs to turn to
 accomplish this? or if it is even possible :)
 We are running cassandra 3.0.3, with
 8Gb heap and a replication factor of 3.
 Thank you in advance!
 Yours sincerely,  Bo
 Madsen
 -- 
 
 
 
 
 
 
 
 
 Jens Rantil
 Backend Developer @ Tink
 Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
 For urgent matters you can reach me at
 +46-708-84 18 32.
 -- 
 
 
 
 
 
 
 
 
 Jens Rantil
 Backend Developer @ Tink
 Tink AB, Wallingatan 5, 111 60
 Stockholm, Sweden
 For urgent matters you can
 reach me at +46-708-84 18 32.

Mime
View raw message