cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wayne <>
Subject Re: Read Latency Degradation
Date Fri, 17 Dec 2010 13:21:27 GMT
We have been testing Cassandra for 6+ months and now have 10TB in 10 nodes
with rf=3. It is 100% real data generated by real code in an almost
production level mode. We have gotten past all our stability issues,
java/cmf issues, etc. etc. now to find the one thing we "assumed" may not be
true. Our current production environment is mysql with extensive
partitioning. We have mysql tables with 3-4 billion records and our query
performance is the same as with 1 million records (< 100ms).

For those of us really trying to manage large volumes of data memory is not
an option in any stretch of the imagination. Our current data volume once
placed within Cassandra ignoring growth should be around 50 TB. We run
manual compaction once a week (absolutely required to keep ss table counts
down) and it is taking a very long amount of time. Now that our nodes are
past 1TB I am worried it will take more than a day. I was hoping everyone
would respond to my posting with something must be wrong, but instead I am
hearing you are off the charts good luck and be patient. Scary to say the
least given our current investment in Cassandra. Is it true/expected that
read latency will get worse in a linear fashion as the ss table size grows?

Can anyone talk me off the fence here? We have 9 MySQL servers that now
serve up 15+TB of data. Based on what we have seen we need 100 Cassandra
nodes with rf=3 to give us good read latency (by keeping the node data sizes
down). The cost/value equation just does not add up.

Thanks in advance for any advice/experience you can provide.

On Fri, Dec 17, 2010 at 5:07 AM, Daniel Doubleday

> On Dec 16, 2010, at 11:35 PM, Wayne wrote:
> > I have read that read latency goes up with the total data size, but to
> what degree should we expect a degradation in performance? What is the
> "normal" read latency range if there is such a thing for a small slice of
> scol/cols? Can we really put 2TB of data on a node and get good read latency
> querying data off of a handful of CFs? Any experience or explanations would
> be greatly appreciated.
> If you really mean 2TB per node I strongly advise you to perform thorough
> testing with real world column sizes and the read write load you expect. Try
> to load test at least with a test cluster / data that represents one
> replication group. I.e. RF=3 -> 3 nodes. And test with the consistency level
> you want to use. Also test ring operations (repair, adding nodes, moving
> nodes) while under expected load/
> Combined with 'a handful of CFs' I would assume that you are expecting a
> considerable write load. You will get massive compaction load and with that
> data size the file system cache will suffer big time. You'll need loads of
> RAM and still ...
> I can only speak about 0.6 but ring management operations will become a
> nightmare and you will have very long running repairs.
> The cluster behavior changes massively with different access patterns (cold
> vs warm data) and data sizes. So you have to understand yours and test it. I
> think most generic load tests are mainly marketing instruments and I believe
> this is especially true for cassandra.
> Don't want to sound negative (I am a believer and don't regret our
> investment) but cassandra is no silver bullet. You really need to know what
> you are doing.
> Cheers,
> Daniel

View raw message