incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janne Jalkanen <Janne.Jalka...@ecyrd.com>
Subject Re: understanding memory footprint
Date Thu, 15 Aug 2013 13:58:38 GMT

Also, if you are using leveled compaction, remember that each SSTable will take a couple of
MB of heap space.  You can tune this by choosing a good sstable_size_in_mb value for those
CFs which are on LCS and contain lots of data.  Default is 5 MB, which is for many cases inadequate,
so most people seem to be happy running with sizes that range from 64 MB and up.  The right
size for you will most probably vary.

/Janne

On Aug 15, 2013, at 06:05 , Aaron Morton <aaron@thelastpickle.com> wrote:

>> "Does the number of column families still significantly impact the memory footprint?
If so, what is the incremental cost of a column family/table?"
> IMHO there would be little difference in memory use for a node with zero data that had
10 CF's and one that had 100 CF's. When you start putting data in the story changes. 
> 
> As Alain said, the number of rows can impact the memory use. In 1.2+ that's less of an
issue, but the index samples are still on heap. In my experience in normal (4Gb to 8GB heap)
this is not an issue until you get into 500+ million rows. 
> 
> The number of CF's is still used when calculating when to flush to disk. If you have
100 cf's the server will flush to disk more frequently than if you have 10. Because it needs
to leave more room for the memtables to grow. 
> 
> The best way to get help on this is provide details on the memory settings, the numbers
of CF's, the total number of rows, and the cache settings. 
> 
> Hope that helps. 
>  
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 13/08/2013, at 9:10 PM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
> 
>> if using 1.2.*, Bloom filters are in native memory so not pressuring your heap, how
many data do you have per node ? If this value is big, you have samples index in the heap
consuming a lot of memory, for sure, and growing as your data per node grow.
>> 
>> Solutions : increase the heap if < 8GB and / or reduce sampling index_interval:
128 to a bigger value (256 - 512) and /or wait for 2.0.* which, of the top of my head, should
move the sampling in native memory allowing heap size to be independent from the data size
per node.
>> 
>> This should alleviate things. Yet these are only guesses since I know almost nothing
about your cluster...
>> 
>> Hope this help somehow.
>> 
>> 
>> 2013/8/12 Robert Coli <rcoli@eventbrite.com>
>> On Mon, Aug 12, 2013 at 11:14 AM, Paul Ingalls <paulingalls@gmail.com> wrote:
>> I don't really need exact numbers, just a rough cost would be sufficient.  I'm running
into memory problems on my cluster, and I'm trying to decide if reducing the number of column
families would be worth the effort.  Looking at the rule of thumb from the wiki entry made
it seem like reducing the number of tables would make a big impact, but I'm running 1.2.8
so not sure if it is still true.
>> 
>> Is there a new rule of thumb?
>>  
>> If you want a cheap/quick measure of how much space partially full memtables are
taking, just nodetool flush and check heap usage before and after?
>> 
>> If you want a cheap/quick measure of how much space empty sstables take in heap,
I think you're out of luck.
>> 
>> =Rob
>> 
>> 
> 


Mime
View raw message