kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Heo <jason.heo....@gmail.com>
Subject Re: tserver died by clock unsync.
Date Fri, 16 Jun 2017 08:59:36 GMT
Hi.

Congrat. Apache Kudu 1.4.0

To prevent tserver from dying accidentally, I've changed LOG(FATAL)
<https://github.com/apache/kudu/blob/1.4.0/src/kudu/server/hybrid_clock.cc#L227>
to LOG(WARNING)

I wanted to know it is safe to continue if ntp_gettime() in GetClockTime
<https://github.com/apache/kudu/blob/1.4.0/src/kudu/server/hybrid_clock.cc#L90>
 returns TIME_ERROR

Could anyone can help me?

Regards,

Jason



2017-06-15 12:40 GMT+09:00 Jason Heo <jason.heo.sde@gmail.com>:

> Hi,
>
> I'm using Apache Kudu 1.4.0
>
> Yesterday, 6 tservers die at the same time. Following message is logged
> for each tserver.
>
>
> F0614 14:58:32.868551 111454 hybrid_clock.cc:227]
>
> Couldn't get the current time: Clock unsynchronized.
>
> Status: Service unavailable:
>
> Error reading clock. Clock considered unsynchronized
>
> We are already using ntpd, and in /var/log/messages, ntpd related message
> is logged.
>
> Jun 14 14:58:38 hostname ntpdate[10231]: step time server ip_addr offset
> -0.000168 sec
>
> We use our own ntp service. I don't know what's the exact reason, but It's
> suspicious that our ntp service is malfunctioned or network is not good
> temporarily.
>
> The problem is that this could happen again and again.
>
> So, I'm considering modifying source code of Kudu from LOG(FATAL) to
> LOG(WARN) so that tserver does not exit on unsync.
>
>   uint64_t now_usec;
>
>   uint64_t error_usec;
>
>   Status s = WalltimeWithError(&now_usec, &error_usec);
>
>   if (PREDICT_FALSE(!s.ok())) {
>
>     LOG(FATAL) << Substitute("Couldn't get the current time: Clock
> unsynchronized. "
>
>         "Status: $0", s.ToString());
>
>   }
>
>
> So, I question is that is it OK modifying LOG(FATAL) to LOG(WARN) of
> above code? and wanted to know this can preventing from dying of tserver
> when clock unsynced?
>
> Thanks.
>
> Jason,
>
> Regard
>

Mime
View raw message