incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Daeschler <david.daesch...@gmail.com>
Subject Re: High CPU usage as of 8pm eastern time
Date Sun, 01 Jul 2012 11:40:24 GMT
More information for others that were affected.

Our installation of java:

[root@inv4 conf]# java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

[root@inv4 conf]# uname -a
Linux inv4 2.6.32-220.4.2.el6.x86_64 #1 SMP Tue Feb 14 04:00:16 GMT
2012 x86_64 x86_64 x86_64 GNU/Linux

Jonathan pointed out a Linux bug that may be related:
https://issues.apache.org/jira/browse/CASSANDRA-4066

In my case only the Java process went nuts, as seems to be the case in
many other reports:
https://bugzilla.mozilla.org/show_bug.cgi?id=769972
http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/

I hope everyone got enough sleep!
- David


On Sun, Jul 1, 2012 at 4:49 AM, Hontvári József Levente
<hontvari@flyordie.com> wrote:
> Thank you for the mail. Same here, but I restarted the affected server
> before I noticed your mail.
>
> It affected both OpenJDK Java 6  (packaged with Ubuntu 10.04) and Oracle
> Java 7 processes. Ubuntu 32 bit servers had no issues, only a 64 bit
> machine.
>
> Likely it is related to the leap second introduced today.
>
>
> On 2012.07.01. 5:11, Mina Naguib wrote:
>>
>> Hi folks
>>
>> Our cassandra (and other java-based apps) started experiencing extremely
>> high CPU usage as of 8pm eastern time (midnight UTC).
>>
>> The issue appears to be related to specific versions of java + linux +
>> ntpd
>>
>> There are many solutions floating around on IRC, twitter, stackexchange,
>> LKML.
>>
>> The simplest one that worked for us is simply to run this command on each
>> affected machine:
>>
>> date; date `date +"%m%d%H%M%C%y.%S"`; date;
>>
>> CPU drop was instantaneous - there was no need to restart the server,
>> ntpd, or any of the affected JVMs.
>>
>>
>>
>>
>
>

Mime
View raw message