cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: SSD vs. HDD
Date Thu, 04 Nov 2010 08:18:52 GMT
> I am having time out errors while reading.
> I have 5 CFs but two CFs with high write/read.
> The data is organized in time series rows, in CF1 the new rows are read
> every 10 seconds and then the whole rows are deleted, While in CF2 the rows
> are read in different time range slices and eventually deleted may be after
> few hours.

So the first thing to do is to confirm what the bottleneck is. If
you're having timeouts on reads, and assuming your not doing reads of
hot-in-cache data so fast that CPU is the bottleneck (and given that
you ask about SSD), the hypothesis then is that you're disk bound due
to seeking.

Observe the node(s) and in particular use "iostat -x -k 1" (or an
equivalent graph) and look at the %util and %avgqu-sz columns to
confirm that you are indeed disk-bound. Unless you're doing large
reads, you will likely see, on average, small reads in amounts that
simply saturate underlying storage, %util at 100% and the avgu-sz will
probably be approaching the level of concurrency of your read traffic.

Now, assuming that is true, the question is why. So:

(1) Are you continually saturating disk or just periodically?
(2) If periodically, does the periods of saturation correlate with
compaction being done by Cassandra (or for that matter something
else)?
(3) What is your data set size relative to system memory? What is your
system memory and JVM heap size? (Relevant because it is important to
look at how much memory the kernel will use for page caching.)

As others have mentioned, the amount of reads done on disk for each
read form the database (assuming data is not in cache) can be affected
by how data is written (e.g., partial row writes etc). That is one
thing that can be addressed, as is re-structuring data to allow
reading more sequentially (if possible). That only helps along one
dimension though - lessening, somewhat, the cost of cold reads. The
gains may be limited and the real problem may be that you simply need
more memory for caching and/or more IOPS from your storage (i.e., more
disks, maybe SSD, etc).

If on the other hand you're normally completely fine and you're just
seeing periods of saturation associated with compaction, this may be
mitigated by software improvements by possibly rate limiting reads
and/or writes during compaction and avoiding buffer cache thrashing.
There's a JIRA ticket for direct I/O
(https://issues.apache.org/jira/browse/CASSANDRA-1470). I don't think
there's a JIRA ticket for rate limiting, but I suspect, since you're
doing time series data, that you're not storing very large values -
and I would expect compaction to be CPU bound rather than being close
to saturate disk.

In either case, please do report back as it's interesting to figure
out what kind of performance issues people are seeing.

-- 
/ Peter Schuller

Mime
View raw message