incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayadev Jayaraman <jdisal...@gmail.com>
Subject Re: What is the ideal value for sstable_size_in_mb when using LeveledCompactionStrategy ?
Date Wed, 18 Sep 2013 20:35:11 GMT
m1.xlarge ( total ephemeral volume size 1.7TB ) is the most widely used
node configuration for Cassandra on EC2 according to this DataStax
presentation : http://www.slideshare.net/mattdennis/cassandra-on-ec2 .

That said, I'm going with 40 for sstable_size_in_mb. My logic is as follows
:

we load around 10+ GB in updates daily using SSTable loader, and with a 24
node cluster, that is about 500 MB per node per day. We have around 5 large
column families and so the amount of newly flushed unleveled SSTable data
per node per column-family per day is around 100 MB on average. Compaction
triggers once every 2 days at least ( when unleveled data per CF exceeds
40*4 = 160 MB ) so there should be about 32 unleveled SSTables per
column-family per node on average, a manageable number. If I reduce
sstable_size_in_mb to 5 MB, I will have 8 times as many SSTable files on
disk (even though all of them will be leveled immediately after flushing)
and I don't know how that impacts I/O performance / number of file-sockets
kept open for serving reads.


On Wed, Sep 18, 2013 at 3:38 PM, Hiller, Dean <Dean.Hiller@nrel.gov> wrote:

> Sorry, bad bad typo…..300G is what I meant.
>
> Cassandra heavily advises to stay under 1T per node or you run into big
> troubles and most people stay under 500G per node.
>
> Later,
> Dean
>
> From: Jayadev Jayaraman <jdisalive@gmail.com<mailto:jdisalive@gmail.com>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> Date: Wednesday, September 18, 2013 1:30 PM
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> Subject: Re: What is the ideal value for sstable_size_in_mb when using
> LeveledCompactionStrategy ?
>
> Thanks for the quick reply. We've already upped the ulimit as high as our
> Linux distro allows us to ( around 1.8 million  ).
>
> I have a follow-up question. I see that the size of individual nodes in
> your use case is quite massive. Does the safe number vary widely based on
> differences in underlying hardware, or would you say from experience that
> something around 50M for medium to large datasets ( with upped
> file-descriptor limits ) is safe for most medium-sized (1 - 5 TB per node)
> to high-end (hundreds of TB) hardware ?
>
>
> On Wed, Sep 18, 2013 at 3:15 PM, Hiller, Dean <Dean.Hiller@nrel.gov
> <mailto:Dean.Hiller@nrel.gov>> wrote:
>  1.  Always in cassandra up your file descriptor limits on linux and even
> in 0.7 that was the recommendation so cassandra could open tons of files
>  2.  We use 50M for our LCS with no performance issues.  We had it 10M on
> our previous with no issues but a huge amount of files of course with our
> 300T per node.
>
> Dean
>
> From: Jayadev Jayaraman <jdisalive@gmail.com<mailto:jdisalive@gmail.com
> ><mailto:jdisalive@gmail.com<mailto:jdisalive@gmail.com>>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org
> ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
<
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
> Date: Wednesday, September 18, 2013 1:02 PM
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
> Subject: What is the ideal value for sstable_size_in_mb when using
> LeveledCompactionStrategy ?
>
> We have set up a 24 node (m1.xlarge nodes, 1.7 TB per node) cassandra
> cluster on Amazon EC2 :
>
> version=1.2.9
> replication factor = 2
> snitch=EC2Snitch
> placement_strategy=NetworkTopologyStrategy (with 12 nodes each in 2
> availability zones)
>
> Background on our use-case :
>
> We plan on using hadoop with sstableloader to load 10GB+ of analytics data
> per day ( 100 million+ row keys, 5 or so columns per day on average.) . We
> have chosen LeveledCompactionStrategy in the hope that it constrains the
> number of SSTables that are read in order to retrieve a sliced-predicate
> for a row. We don't want too many file-sockets ( > 1000) open to SSTables
> by the Cassandra JVM as this has caused us network / unreachability issues
> before. We faced this when we were on cassandra 0.8.9 and we were using
> SizeTieredCompactionStrategy and in order to mitigate this, we ran minor
> compaction daily and major compaction semi-regularly to ensure as few
> SSTable files as possible on disk.
>
>
>
>
>
> If we use LeveledCompactionStrategy with a small value for
> sstable_size_in_mb ( default = 5 MB ) , wouldn't that result in a very
> large number of SSTable files on disk ? How does that affect the number of
> file-sockets open (reading the docs, I get the impression that the number
> of SSTable seeks per query is reduced by a large margin) ? But if we use a
> larger value for sstable_size_in_mb, say around 200 MB, there will be 800
> MB of small uncompacted SSTables on disk per column-family to which there
> will inevitably be file-sockets open.
>
> All in all, can someone help us figure out what we should set the value of
> sstable_size_in_mb to ? I figure it's not a very good idea to set it to a
> larger value but I don't know how things perform if we set it to a small
> value. Do we have to run major compaction regularly in this case too ?
>
> Thanks
> Jayadev
>
>
>
>

Mime
View raw message