cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: Lot of pending tasks for writes
Date Tue, 12 Apr 2011 20:47:08 GMT
> I am just running simple test in 6 node cassandra 4 GB heap, 96 GB RAM and
> 12 core per host. I am inserting 1M rows with avg col size of 250k. I keep
> getting "Dropped mutation" messages in logs. Not sure how to troubleshoot or
> tune it.

Average col size of 250k - that sounds to me like you're almost
certainly going to be bottlenecking on disk I/O.

Saturating your "active" in the mutation stage and building up pending
is consistent with simply writing faster than writes can be handled.
At first I was skeptical and figured maybe something was wrong, but
upon re-reading and spotting your 250k column size - it's really easy
to have a stress client saturate nodes with data sizes that large.

The first thing I would do is to just look at what's going on on the
system. For example, just run "iostat -x -k 1" on the machines and see
whether you're completely disk bound or not. I suspect you are, and
that the effects you're seeing is simply the result of that.

However that would depend on how many mutations per second you're
actually sending. But if you're using out-of-the-box without
rate limiting and using a column size of 250k, I am not at all
surprised that you're easily able to saturate your nodes.

/ Peter Schuller

View raw message