Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B20859CB7 for ; Thu, 26 Jan 2012 05:53:17 +0000 (UTC) Received: (qmail 82008 invoked by uid 500); 26 Jan 2012 05:53:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81719 invoked by uid 500); 26 Jan 2012 05:53:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 81706 invoked by uid 99); 26 Jan 2012 05:53:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jan 2012 05:53:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of drwoland@gmail.com designates 74.125.83.44 as permitted sender) Received: from [74.125.83.44] (HELO mail-ee0-f44.google.com) (74.125.83.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jan 2012 05:52:56 +0000 Received: by eekc1 with SMTP id c1so49797eek.31 for ; Wed, 25 Jan 2012 21:52:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=o1J/AiSgmPk0a5pnvqEjLTO7Qegeit5uxVbXY98Oicw=; b=olaSpPH44vM2+3pn790SS0EQ42IN1M+IL5D9ruEadon4S1+dz/EBIBcS5v4+n+uyFK SPkIZtR8RF0mOJf1Vj142FGkrnoUs6uVAD4HyqVh1UrTSQLKZ8t0t91bG0dKc03CbDcY wLwQBF9brXWvCovg5f7sMOSaqSKhVLn36OPG4= MIME-Version: 1.0 Received: by 10.14.17.91 with SMTP id i67mr277757eei.10.1327557153593; Wed, 25 Jan 2012 21:52:33 -0800 (PST) Sender: drwoland@gmail.com Received: by 10.213.35.68 with HTTP; Wed, 25 Jan 2012 21:52:33 -0800 (PST) In-Reply-To: <30C7D8E3-3C2E-40C5-8629-302DEBDC480A@thelastpickle.com> References: <4F20464B.9060308@hiramoto.org> <4F20491F.3070304@hiramoto.org> <6F1D54DD1FE19B408496215469822A8701F060B376D3@OCDP-ERFMMBX03.ERF.THOMSON.COM> <30C7D8E3-3C2E-40C5-8629-302DEBDC480A@thelastpickle.com> Date: Wed, 25 Jan 2012 21:52:33 -0800 X-Google-Sender-Auth: Om_nDcmvu2kg5WIE-LbUdEhp0e4 Message-ID: Subject: Re: Restart cassandra every X days? From: Mike Panchenko To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e6542d288f189904b767fc5d --0016e6542d288f189904b767fc5d Content-Type: text/plain; charset=ISO-8859-1 There are two relevant bugs (that I know of), both resolved in somewhat recent versions, which make somewhat regular restarts beneficial https://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak in GCInspector, fixed in 0.7.9/0.8.5) https://issues.apache.org/jira/browse/CASSANDRA-2252 (heap fragmentation due to the way memtables used to be allocated, refactored in 1.0.0) Restarting daily is probably too frequent for either one of those problems. We usually notice degraded performance in our ancient cluster after ~2 weeks w/o a restart. As Aaron mentioned, if you have plenty of disk space, there's no reason to worry about "cruft" sstables. The size of your active set is what matters, and you can determine if that's getting too big by watching for iowait (due to reads from the data partition) and/or paging activity of the java process. When you hit that problem, the solution is to 1. try to tune your caches and 2. add more nodes to spread the load. I'll reiterate - looking at raw disk space usage should not be your guide for that. "Forcing" a gc generally works, but should not be relied upon (note "suggest" in http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#gc()). It's great news that 1.0 uses a better mechanism for releasing unused sstables. nodetool compact triggers a "major" compaction and is no longer a recommended by datastax (details here http://www.datastax.com/docs/1.0/operations/tuning#tuning-compaction bottom of the page). Hope this helps. Mike. On Wed, Jan 25, 2012 at 5:14 PM, aaron morton wrote: > That disk usage pattern is to be expected in pre 1.0 versions. Disk usage > is far less interesting than disk free space, if it's using 60 GB and there > is 200GB thats ok. If it's using 60Gb and there is 6MB free thats a problem. > > In pre 1.0 the compacted files are deleted on disk by waiting for the JVM > do decide to GC all remaining references. If there is not enough space (to > store the total size of the files it is about to write or compact) on disk > GC is forced and the files are deleted. Otherwise they will get deleted at > some point in the future. > > In 1.0 files are reference counted and space is freed much sooner. > > With regard to regular maintenance, node tool cleanup remvos data from a > node that it is no longer a replica for. This is only of use when you have > done a token move. > > I would not recommend a daily restart of the cassandra process. You will > lose all the run time optimizations the JVM has made (i think the mapped > files pages will stay resident). As well as adding additional entropy to > the system which must be repaired via HH, RR or nodetool repair. > > If you want to see compacted files purged faster the best approach would > be to upgrade to 1.0. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 26/01/2012, at 9:51 AM, R. Verlangen wrote: > > In his message he explains that it's for " Forcing a GC ". GC stands for > garbage collection. For some more background see: > http://en.wikipedia.org/wiki/Garbage_collection_(computer_science) > > Cheers! > > 2012/1/25 > >> Karl, >> >> Can you give a little more details on these 2 lines, what do they do? >> >> java -jar cmdline-jmxclient-0.10.3.jar - localhost:8080 >> java.lang:type=Memory gc >> >> Thank you, >> Mike >> >> -----Original Message----- >> From: Karl Hiramoto [mailto:karl@hiramoto.org] >> Sent: Wednesday, January 25, 2012 12:26 PM >> To: user@cassandra.apache.org >> Subject: Re: Restart cassandra every X days? >> >> >> On 01/25/12 19:18, R. Verlangen wrote: >> > Ok thank you for your feedback. I'll add these tasks to our daily >> > cassandra maintenance cronjob. Hopefully this will keep things under >> > controll. >> >> I forgot to mention that we found that Forcing a GC also cleans up some >> space. >> >> >> in a cronjob you can do this with >> http://crawler.archive.org/cmdline-jmxclient/ >> >> >> my cronjob looks more like >> >> nodetool repair >> nodetool cleanup >> nodetool compact >> java -jar cmdline-jmxclient-0.10.3.jar - localhost:8080 >> java.lang:type=Memory gc >> >> -- >> Karl >> >> This email was sent to you by Thomson Reuters, the global news and >> information company. Any views expressed in this message are those of the >> individual sender, except where the sender specifically states them to be >> the views of Thomson Reuters. >> > > > --0016e6542d288f189904b767fc5d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable There are two relevant bugs (that I know of), both resolved in somewhat rec= ent versions, which make somewhat regular restarts beneficial

http= s://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak in GCInsp= ector, fixed in 0.7.9/0.8.5)
https= ://issues.apache.org/jira/browse/CASSANDRA-2252 (heap fragmentation due= to the way memtables used to be allocated, refactored in 1.0.0)

Restarting daily is probably too frequent for either one of = those problems. We usually notice degraded performance in our ancient clust= er after ~2 weeks w/o a restart.

As Aaron mentione= d, if you have plenty of disk space, there's no reason to worry about &= quot;cruft" sstables. The size of your active set is what matters, and= you can determine if that's getting too big by watching for iowait (du= e to reads from the data partition) and/or paging activity of the java proc= ess. When you hit that problem, the solution is to 1. try to tune your cach= es and 2. add more nodes to spread the load. I'll reiterate - looking a= t raw disk space usage should not be your guide for that.

"Forcing" a gc generally works, but should no= t be relied upon (note "suggest" in=A0http://docs.oracle.com= /javase/6/docs/api/java/lang/System.html#gc()). It's great news tha= t 1.0 uses a better mechanism for releasing unused sstables.

nodetool compact triggers a "major" compactio= n and is no longer a recommended by datastax (details here=A0http://w= ww.datastax.com/docs/1.0/operations/tuning#tuning-compaction bottom of = the page).

Hope this helps.

Mike.

On Wed, Jan 25, 2012 at 5:14 PM, aaron mo= rton <aaron= @thelastpickle.com> wrote:
That dis= k usage pattern is to be expected in pre 1.0 versions. Disk usage is far le= ss interesting than disk free space, if it's using 60 GB and there is 2= 00GB thats ok. If it's using 60Gb and there is 6MB free thats a problem= .

In pre 1.0 the compacted files are deleted on disk by waitin= g for the JVM do decide to GC all remaining references. If there is not eno= ugh space (to store the total size of the files it is about to write or com= pact) on disk GC is forced and the files are deleted. Otherwise they will g= et deleted at some point in the future.=A0

In 1.0 files are reference counted and space is freed m= uch sooner.=A0

With regard to regular maintenance,= node tool cleanup remvos data from a node that it is no longer a replica f= or. This is only of use when you have done a token move.=A0

I would not recommend a daily restart of the cassandra = process. You will lose all the run time optimizations the JVM has made (i t= hink the mapped files pages will stay resident). As well as adding addition= al entropy to the system which must be repaired via HH, RR or nodetool repa= ir.=A0

If you want to see compacted files purged faster the be= st approach would be to upgrade to 1.0.=A0

Hope th= at helps.=A0

<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Developer
@aaronmorton=

On 26/01/2012, at 9:51 AM, R. Verlangen wrote:

In his message he explains that it's for " Forcing a GC=A0". GC stands for garbage collection. For s= ome more background see:=A0 http://en.wikipedia.org/wiki/Garbage_collection_(comp= uter_science)=A0

Cheers!

2012/1/25 <mike.li@thomsonreuters.com>
Karl,

Can you give a little more details on these 2 lines, what do they do?

java -jar cmdline-jmxclient-0.10.3.jar - localhost:8080
java.lang:type=3DMemory gc

Thank you,
Mike

-----Original Message-----
From: Karl Hiramoto [mailto:karl@hiramoto.org]
Sent: Wednesday, January 25, 2012 12:26 PM
To: user@cas= sandra.apache.org
Subject: Re: Restart cassandra every X days?


On 01/25/12 19:18, R. Verlangen wrote:
> Ok thank you for your feedback. I'll add these tasks to our daily<= br> > cassandra maintenance cronjob. Hopefully this will keep things under > controll.

I forgot to mention that we found that Forcing a GC also cleans up some
space.


in a cronjob you can do this with
http://crawler.archive.org/cmdline-jmxclient/


my cronjob looks more like

nodetool repair
nodetool cleanup
nodetool compact
java -jar cmdline-jmxclient-0.10.3.jar - localhost:8080
java.lang:type=3DMemory gc

--
Karl

This email was sent to you by Thomson Reuters, the global news = and information company. Any views expressed in this message are those of t= he individual sender, except where the sender specifically states them to b= e the views of Thomson Reuters.



--0016e6542d288f189904b767fc5d--