cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "R. Verlangen" <ro...@us2.nl>
Subject Re: Restart cassandra every X days?
Date Sat, 28 Jan 2012 12:53:42 GMT
Ok, seems that it's clear what I should do next ;-)

2012/1/28 aaron morton <aaron@thelastpickle.com>

> There are no blockers to upgrading to 1.0.X.
>
> A
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 28/01/2012, at 7:48 AM, R. Verlangen wrote:
>
> Ok. Seems that an upgrade might fix these problems. Is Cassandra 1.x.x
> stable enough to upgrade for, or should we wait for a couple of weeks?
>
> 2012/1/27 Edward Capriolo <edlinuxguru@gmail.com>
>
>> I would not say that issuing restart after x days is a good idea. You are
>> mostly developing a superstition. You should find the source of the
>> problem. It could be jmx or thrift clients not closing connections. We
>> don't restart nodes on a regiment they work fine.
>>
>>
>> On Thursday, January 26, 2012, Mike Panchenko <m@mihasya.com> wrote:
>> > There are two relevant bugs (that I know of), both resolved in somewhat
>> recent versions, which make somewhat regular restarts beneficial
>> > https://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak in
>> GCInspector, fixed in 0.7.9/0.8.5)
>> > https://issues.apache.org/jira/browse/CASSANDRA-2252 (heap
>> fragmentation due to the way memtables used to be allocated, refactored in
>> 1.0.0)
>> > Restarting daily is probably too frequent for either one of those
>> problems. We usually notice degraded performance in our ancient cluster
>> after ~2 weeks w/o a restart.
>> > As Aaron mentioned, if you have plenty of disk space, there's no reason
>> to worry about "cruft" sstables. The size of your active set is what
>> matters, and you can determine if that's getting too big by watching for
>> iowait (due to reads from the data partition) and/or paging activity of the
>> java process. When you hit that problem, the solution is to 1. try to tune
>> your caches and 2. add more nodes to spread the load. I'll reiterate -
>> looking at raw disk space usage should not be your guide for that.
>> > "Forcing" a gc generally works, but should not be relied upon (note
>> "suggest" in
>> http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#gc()).
>> It's great news that 1.0 uses a better mechanism for releasing unused
>> sstables.
>> > nodetool compact triggers a "major" compaction and is no longer a
>> recommended by datastax (details here
>> http://www.datastax.com/docs/1.0/operations/tuning#tuning-compactionbottom of the
page).
>> > Hope this helps.
>> > Mike.
>> > On Wed, Jan 25, 2012 at 5:14 PM, aaron morton <aaron@thelastpickle.com>
>> wrote:
>> >
>> > That disk usage pattern is to be expected in pre 1.0 versions. Disk
>> usage is far less interesting than disk free space, if it's using 60 GB and
>> there is 200GB thats ok. If it's using 60Gb and there is 6MB free thats a
>> problem.
>> > In pre 1.0 the compacted files are deleted on disk by waiting for the
>> JVM do decide to GC all remaining references. If there is not enough space
>> (to store the total size of the files it is about to write or compact) on
>> disk GC is forced and the files are deleted. Otherwise they will get
>> deleted at some point in the future.
>> > In 1.0 files are reference counted and space is freed much sooner.
>> > With regard to regular maintenance, node tool cleanup remvos data from
>> a node that it is no longer a replica for. This is only of use when you
>> have done a token move.
>> > I would not recommend a daily restart of the cassandra process. You
>> will lose all the run time optimizations the JVM has made (i think the
>> mapped files pages will stay resident). As well as adding additional
>> entropy to the system which must be repaired via HH, RR or nodetool repair.
>> > If you want to see compacted files purged faster the best approach
>> would be to upgrade to 1.0.
>> > Hope that helps.
>> > -----------------
>> > Aaron Morton
>> > Freelance Developer
>> > @aaronmorton
>> > http://www.thelastpickle.com
>> > On 26/01/2012, at 9:51 AM, R. Verlangen wrote:
>> >
>> > In his message he explains that it's for " Forcing a GC ". GC stands
>> for garbage collection. For some more background see:
>> http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)
>> > Cheers!
>> >
>> > 2012/1/25 <mike.li@thomsonreuters.com>
>> >
>> > Karl,
>> >
>> > Can you give a little more details on these 2 lines, what do they do?
>> >
>> > java -jar cmdline-jmxclient-0.10.3.jar - localhost:8080
>> > java.lang:type=Memory gc
>> >
>> > Thank you,
>> > Mike
>> >
>> > -----Original Message-----
>> > From: Karl Hiramoto [mailto:karl@hiramoto.org]
>> > Sent: Wednesday, January 25, 2012 12:26 PM
>> > To: user@cassandra.apache.org
>> > Subject: Re: Restart cassandra every X days?
>> >
>> >
>> > On 01/25/12 19:18, R. Verlangen wrote:
>> >> Ok thank you for your feedback. I'll add these tasks to our daily
>> >> cassandra maintenance cronjob. Hopefully this will keep things under
>> >> controll.
>> >
>> > I forgot to mention that we found that Forcing a GC also cleans up some
>> > space.
>> >
>> >
>> > in a cronjob you can do this with
>> > http://crawler.archive.org/cmdline-jmxclient/
>> >
>> >
>> > my cron
>>
>
>
>

Mime
View raw message