Thanks for the detailed answers Dan, what you said makes sense. I think my biggest worry right now is making the correct preditions of my data storage space based on the measurements with the current cluster. Other than that I should be fairly comfortable with the rest of the HW specs.

Thanks for the observation Mohit, I'll keep a closer eye to this disk parameter which I do see in the specs all the time.

Todd, your link explains questions I have had for quite some time .... I have found that indeed I am dominated by metadata like one of the example shows.

Since we're on this subject, I want to ask you guys another question. I have my monitoring data sources within an enclosed network, so my Cassandra cluster will also be in that enclosed network (by enclosed network I mean any communication or data transfer in and out of the network must go through a gateway). The problem is I need to make the data available outside!
Do any of you guys have some suggestions for doing that? My first thought was to have an internal 3 node cluster taking in the insertion load and then, in the period of low load do a major compaction and then ship the data out to an external Cassandra node used only for reading. This outside node would have to have a lot of disk (hold the data for 1 year) and be optimised for reading - I was thinking of having an SSD caching layer between my bulk storage and Cassandra. Only hot data will go in this layer....somehow!

So my questions:
1) Is my method unheard of or does it sound reasonable?
2) What is the best way to transfer data from the cluster inside the enclosed network to the node outside? I heard some time in the past that there is a tool that does bulk transfers of data but I'm not sure how that can be done...a script that calls this tool on a certain trigger..........any ideas?
3) Is this intermediate SSD cache thing doable...or I should just stick to the normal RAID array of disks and the indexes and in memory caching of columns that Cassandra offers?

Cheers,
Alex

On Tue, Oct 25, 2011 at 9:06 PM, Todd Burruss <bburruss@expedia.com> wrote:
This may help determining your data storage requirements ...

http://btoddb-cass-storage.blogspot.com/



On 10/25/11 11:22 AM, "Mohit Anchlia" <mohitanchlia@gmail.com> wrote:

>On Tue, Oct 25, 2011 at 11:18 AM, Dan Hendry <dan.hendry.junk@gmail.com>
>wrote:
>>> 2. ... So I am going to use rotational disk for the commit log and an
>>>SSD
>>> for data. Does this make sense?
>>
>>
>>
>> Yes, just keep in mind however that the primary characteristic of SSDs
>>is
>> lower seek times which translates into faster random access. We have a
>> similar Cassandra use case (time series data and comparable volumes) and
>> decided the random read performance boost (unquantified in our case to
>>be
>> fair) was not worth the price and we went with more, larger, cheaper
>>7.2k
>> HDDs.
>>
>>
>>
>>> 3. What's the best way to find out how big my commitlog disk and my
>>>data
>>> disk has to be? The Cassandra hardware page says the Commitlog disk
>>> shouldn't be big but still I need to choose a size!
>>
>>
>>
>> As of Cassandra 1.0, the commit log has an explicit size bound
>>(defaulting
>> to 4GB I believe). In 0.8, I dont think I have ever seen my commit log
>>grow
>> beyond that point but the limit should be the ammount of data you insert
>> within the maximum CF timed flush period (³memtable_flush_after²
>>parameter,
>> to be safe, maximumum across all CFs). Any modern drive should be
>> sufficient. As for the size of your data disks, that is largely
>>application
>> dependent, and you should be able to judge best based on your currnet
>> cluster.
>>
>>
>>
>>> 4. I also noticed RAID 0 configuration is recommended for the data file
>>> directory. Can anyone explain why?
>>
>>
>>
>> In comparison to RAID1/RAID1+0? For any RF > 1, Cassadra already takes
>>care
>> of redundancy by replicating the data across multiple nodes. Your
>> applications choice of replication factor and read/write consistencies
>> should be specified to tollerate a node failing (for any reason: disk
>> failure, network failure, a disgruntled employee taking a sledge hammer
>>to
>> the box, etc). As such, what is the point of waisting your disks
>>duplicating
>> data on a single machine to minimize the chances of one particular type
>>of
>> failure when it should not matter anyways?
>
>It all boils down to operations cost vs hardware cost. Also consider
>MTBF and how equipped you are to handle disk failures which are more
>common than others.
>>
>>
>>
>> Dan
>>
>>
>>
>> From: Alexandru Sicoe [mailto:adsicoe@gmail.com]
>> Sent: October-25-11 8:23
>> To: user@cassandra.apache.org
>> Subject: Cassandra cluster HW spec (commit log directory vs data file
>> directory)
>>
>>
>>
>> Hi everyone,
>>
>> I am currently in the process of writing a hardware proposal for a
>>Cassandra
>> cluster for storing a lot of monitoring time series data. My workload is
>> write intensive and my data set is extremely varied in types of
>>variables
>> and insertion rate for these variables (I will have to handle an order
>>of 2
>> million variables coming in, each at very different rates - the
>>majority of
>> them will come at very low rates but there are many that will come at
>>higher
>> rates constant rates and a few coming in with huge spikes in rates).
>>These
>> variables correspond to all basic C++ types and arrays of these types.
>>The
>> highest insertion rates are received for basic types, out of which U32
>> variables seem to be the most prevalent (e.g. I recorded 2 million U32
>>vars
>> were inserted in 8 mins of operation while 600.000 doubles and 170.000
>> strings were inserted during the same time. Note this measurement was
>>only
>> for a subset of the total data currently taken in).
>>
>> At the moment I am partitioning the data in Cassandra in 75 CFs (each CF
>> corresponds to a logical partitioning of the set of variables mentioned
>> before - but this partitioning is not related with the amount of data or
>> rates...it is somewhat random). These 75 CFs account for ~1 million of
>>the
>> variables I need to store. I have a 3 node Cassandra 0.8.5 cluster (each
>> node is a 4 real core with 4 GB RAM and split commit log directory and
>>data
>> file directory between two RAID arrays with HDDs). I can handle the
>>load in
>> this configuration but the average CPU usage of the Cassandra nodes is
>> slightly above 50%. As I will need to add 12 more CFs (corresponding to
>> another ~ 1 million variables) plus potentially other data later, it is
>> clear that I need better hardware (also for the retrieval part).
>>
>> I am looking at Dell servers (Power Edge etc)
>>
>> Questions:
>>
>> 1. Is anyone using Dell HW for their Cassandra clusters? How do they
>>behave?
>> Anybody care to share their configurations or tips for buying, what to
>>avoid
>> etc?
>>
>> 2. Obviously I am going to keep to the advice on the
>> http://wiki.apache.org/cassandra/CassandraHardware and split the
>>commmitlog
>> and data on separate disks. I was going to use SSD for commitlog but
>>then
>> did some more research and found out that it doesn't make sense to use
>>SSDs
>> for sequential appends because it won't have a performance advantage
>>with
>> respect to rotational media. So I am going to use rotational disk for
>>the
>> commit log and an SSD for data. Does this make sense?
>>
>> 3. What's the best way to find out how big my commitlog disk and my data
>> disk has to be? The Cassandra hardware page says the Commitlog disk
>> shouldn't be big but still I need to choose a size!
>>
>> 4. I also noticed RAID 0 configuration is recommended for the data file
>> directory. Can anyone explain why?
>>
>> Sorry for the huge email.....
>>
>> Cheers,
>> Alex
>>
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com
>> Version: 9.0.920 / Virus Database: 271.1.1/3972 - Release Date: 10/24/11
>> 14:35:00