incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Golick <jamesgol...@gmail.com>
Subject Never ending compaction
Date Wed, 23 Jun 2010 05:08:28 GMT
We had to take a node down for an upgrade last night. When we brought it
back online in the morning, it got slammed by HH data all day so badly that
it was compacting near constantly, and the pending compactions pool was
piling up. I shut most of the writes down to let things catch up, which they
mostly have, but in an effort to minimize downtime of the component that
relies on cassandra, I restarted the reading and writing with 4 pending
compactions.

Now, the writing is taking place quickly enough that compactions just keep
queueing up. That basically means that at this pace, compactions will
*never* complete. And compactions are expensive. They essentially make a
node useless. So, we're left with 3/4 of a cluster, since we only have 4
nodes.

Since then, another node in the cluster has started queueing up compactions.

This is on pretty beefy hardware, too:

2 x E5620, 24GB, 2 x 15kRPM SAS disks in RAID1 for data, and 1 x 7200RPM
SATA for commit logs.

I guess we need more nodes? But, we only have about 80GB total per node,
which doesn't really seem like that much for that kind of hardware?

- James

Mime
View raw message