incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Szerdahelyi <andras.szerdahe...@ignitionone.com>
Subject Re: Cassandra freezes
Date Wed, 20 Mar 2013 12:41:01 GMT
2. Upping index_interval from 128 to 512 (this seemed to reduce our memory
usage significantly!!!)


I'd be very careful with that as a one-stop improvement solution for two
reasons AFAIK
1) you have to rebuild stables ( not an issue if you are evaluating, doing
test writes.. Etc, not so much in production )
2) it can affect reads ( number of sstable reads to serve a read )
especially if your key/row cache is ineffective

On 20/03/13 13:34, "Hiller, Dean" <Dean.Hiller@nrel.gov> wrote:

>Also, look at the cassandra logs.  I bet you see the typicalÅ blah blah is
>at 0.85, doing memory cleanup which is not exactly GC but cassandra memory
>managementÅ ..and of course, you have GC on top of that.
>
>If you need to get your memory down, there are multiple ways
>1. Switching size tiered compaction to leveled compaction(with 1 billion
>narrow rows, this helped us quite a bit)
>2. Upping index_interval from 128 to 512 (this seemed to reduce our memory
>usage significantly!!!)
>3. Just add more nodes as moving the rows to other servers reduces memory
>from #1 and #2 above since the server would have less rows
>
>Later,
>Dean
>
>On 3/20/13 6:29 AM, "Andras Szerdahelyi"
><andras.szerdahelyi@ignitionone.com> wrote:
>
>>
>>I'd say GC. Please fill in form CASS-FREEZE-001 below and get back to us
>>:-) ( sorry )
>>
>>How big is your JVM heap ? How many CPUs ?
>>Garbage collection taking long ? ( look for log lines from GCInspector)
>>Running out of heap ? ( "heap is .. full" log lines )
>>Any tasks backing up / being dropped ? ( nodetool tpstats and ".. dropped
>>in last .. ms" log lines )
>>Are writes really slow? ( nodetool cfhistograms Keyspace ColumnFamily )
>>
>>How much is lots of data? Wide or skinny rows? Mutations/sec ?
>>Which Compaction Strategy are you using? Output of show schema (
>>cassandra-cli ) for the relevant Keyspace/CF might help as well
>>
>>What consistency are you doing your writes with ? I assume ONE or ANY if
>>you have a single node.
>>
>>What are the values for these settings in cassandra.yaml
>>
>>memtable_total_space_in_mb:
>>memtable_flush_writers:
>>memtable_flush_queue_size:
>>compaction_throughput_mb_per_sec:
>>
>>concurrent_writes:
>>
>>
>>
>>Which version of Cassandra?
>>
>>
>>
>>Regards,
>>Andras
>>
>>From:  Joel Samuelsson <samuelsson.joel@gmail.com>
>>Reply-To:  "user@cassandra.apache.org" <user@cassandra.apache.org>
>>Date:  Wednesday 20 March 2013 13:06
>>To:  "user@cassandra.apache.org" <user@cassandra.apache.org>
>>Subject:  Cassandra freezes
>>
>>
>>Hello,
>>
>>I've been trying to load test a one node cassandra cluster. When I add
>>lots of data, the Cassandra node freezes for 4-5 minutes during which
>>neither reads nor writes are served.
>>During this time, Cassandra takes 100% of a single CPU core.
>>My initial thought was that this was Cassandra flushing memtables to the
>>disk, however, the disk i/o is very low during this time.
>>Any idea what my problem could be?
>>I'm running in a virtual environment in which I have no control of
>>drives.
>>So commit log and data directory is (probably) on the same drive.
>>
>>Best regards,
>>Joel Samuelsson
>>
>


Mime
View raw message