cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Stupp (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-11547) Add background thread to check for clock drift
Date Fri, 22 Apr 2016 12:10:13 GMT


Robert Stupp commented on CASSANDRA-11547:

I'm in favor of having this patch. Bad things happen in the wild and I agree to the argument
that having something that gives an operator an indication that the wall clock somehow went
too fast or too slow can definitely explain things that would otherwise be considered as "magic"
or "strange behavior".
Making it a bit more configurable to the points mentioned by Sylvain would be a good thing.

> Add background thread to check for clock drift
> ----------------------------------------------
>                 Key: CASSANDRA-11547
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jason Brown
>            Assignee: Jason Brown
>            Priority: Minor
>              Labels: clocks, time
> The system clock has the potential to drift while a system is running. As a simple way
to check if this occurs, we can run a background thread that wakes up every n seconds, reads
the system clock, and checks to see if, indeed, n seconds have passed. 
> * If the clock's current time is less than the last recorded time (captured n seconds
in the past), we know the clock has jumped backward.
> * If n seconds have not elapsed, we know the system clock is running slow or has moved
backward (by a value less than n)
> * If (n + a small offset) seconds have elapsed, we can assume we are within an acceptable
window of clock movement. Reasons for including an offset are the clock checking thread might
not have been scheduled on time, or garbage collection, and so on.
> * If the clock is greater than (n + a small offset) seconds, we can assume the clock
jumped forward.
> In the unhappy cases, we can write a message to the log and increment some metric that
the user's monitoring systems can trigger/alert on.

This message was sent by Atlassian JIRA

View raw message