cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Brown (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11547) Add background thread to check for clock drift
Date Fri, 22 Apr 2016 14:03:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253981#comment-15253981
] 

Jason Brown edited comment on CASSANDRA-11547 at 4/22/16 2:02 PM:
------------------------------------------------------------------

bq. [~snazy] Making it a bit more configurable

I thought it was reasonably configurable :). I'd be happy to add more if we think it's reasonable.

bq. [~tjake] I don't think we can reliably detect minor differences

I agree, and that's why the patch wakes up every five minutes, iirc; the wake period is comfigurable.
With this patch, we're not trying catch things at the smallest size, a la jHiccup, but really
just want to catch things after large enough time distances. The defaults were intended to
work around/with "large" GC pauses, and we can change the wording for log messages to include
references to that; but at the end of the day if you've got 5 minute GC pauses, you've got
problems, anyway.


was (Author: jasobrown):
bq. [~snazy] Making it a bit more configurable

I thought it was reasonably configurable :). I'd be happy to add more if think it's reasonable.

bq. [~tjake] I don't think we can reliably detect minor differences

I agree, and that's why the patch wakes up every five minutes, iirc; the wake period is comfigurable.
With this patch, we're not trying catch things at the smallest size, a la jHiccup, but really
just want to catch things after large enough time distances. The defaults were intended to
work around/with "large" GC pauses, and we can change the wording for log messages to include
references to that; but at the end of the day if you've got 5 minute GC pauses, you've got
problems, anyway.

> Add background thread to check for clock drift
> ----------------------------------------------
>
>                 Key: CASSANDRA-11547
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11547
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jason Brown
>            Assignee: Jason Brown
>            Priority: Minor
>              Labels: clocks, time
>
> The system clock has the potential to drift while a system is running. As a simple way
to check if this occurs, we can run a background thread that wakes up every n seconds, reads
the system clock, and checks to see if, indeed, n seconds have passed. 
> * If the clock's current time is less than the last recorded time (captured n seconds
in the past), we know the clock has jumped backward.
> * If n seconds have not elapsed, we know the system clock is running slow or has moved
backward (by a value less than n)
> * If (n + a small offset) seconds have elapsed, we can assume we are within an acceptable
window of clock movement. Reasons for including an offset are the clock checking thread might
not have been scheduled on time, or garbage collection, and so on.
> * If the clock is greater than (n + a small offset) seconds, we can assume the clock
jumped forward.
> In the unhappy cases, we can write a message to the log and increment some metric that
the user's monitoring systems can trigger/alert on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message