cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anubhav Kale (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10580) On dropped mutations, more details should be logged.
Date Sat, 12 Dec 2015 18:05:46 GMT


Anubhav Kale commented on CASSANDRA-10580:

Thanks for the interesting suggestion. Actually I considered going down that route when I
started working on it, but I just wasn't sure what the rationale / design philosophy behind
adding new metrics was therefore took a simpler route. Glad to see your feedback.

I have attached 10580-metrics.patch and will open a separate JIRA for doing this on a CF basis.
I am using ApproximateTime class wherever its not taking part in decision of dropping the
mutation and simply used for logging. I hope that makes sense.

I can clean up the methods in MessagingService a bit more if you like (couple of them are
printing the same message). I wanted to send this out first to make sure I was on the right

Also, a question: It appears that Timer.Update appends entries to the metric (which is what
we want). Do you know at what point it starts dropping new appends / starts giving up ? I
wonder if there is a huge number of dropped mutations will the timeTaken metric mess up ?

To make this work for CF, I will probably pass the mutation to MessagingService.LogDroppedMessages
(maybe through an overload) and update the metrics on appropriate CF. Does that make sense

If this change looks good, I am more inclined towards making this work for CF before making
up patches for old branches. Let me know if that's okay.

Appreciate your time and feedback !

> On dropped mutations, more details should be logged.
> ----------------------------------------------------
>                 Key: CASSANDRA-10580
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Coordination
>         Environment: Production
>            Reporter: Anubhav Kale
>            Assignee: Anubhav Kale
>            Priority: Minor
>             Fix For: 3.2, 2.2.x
>         Attachments: 10580-Metrics.patch, 10580.patch, CASSANDRA-10580-Head.patch, Trunk.patch
> In our production cluster, we are seeing a large number of dropped mutations. At a minimum,
we should print the time the thread took to get scheduled thereby dropping the mutation (We
should also print the Message / Mutation so it helps in figuring out which column family was
affected). This will help find the right tuning parameter for write_timeout_in_ms. 
> The change is small and is in and I will submit
a patch shortly.

This message was sent by Atlassian JIRA

View raw message