incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei Zhu <wz1...@yahoo.com>
Subject Re: compaction throughput rate not even close to 16MB
Date Wed, 24 Apr 2013 22:39:07 GMT
Same here. We disable the throttling and our disk and CPU usage both low (< 10%) and still
takes hours for LCS compaction to finish after a repair. For this cluster, we don't delete
any data, so we can rule out tombstones. Not sure what is holding compaction back. My observation
is that for the LCS which involves large number of SSTables (since we set SSTable size too
small at 10M and sometimes one compactions involves up to 10 G of data = 1000 SSTables), the
throughout put is smaller. So my theory is that open/close file handlers have substantial
impact on the throughput. 

By the way, we are on SSD.

-Wei

________________________________
 From: "Hiller, Dean" <Dean.Hiller@nrel.gov>
To: "user@cassandra.apache.org" <user@cassandra.apache.org> 
Sent: Wednesday, April 24, 2013 1:37 PM
Subject: Re: compaction throughput rate not even close to 16MB
 

Thanks much!!!  Better to hear at least one other person sees the same thing ;).  Sometimes
these posts just go silent.

Dean

From: Edward Capriolo <edlinuxguru@gmail.com<mailto:edlinuxguru@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Wednesday, April 24, 2013 2:33 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: compaction throughput rate not even close to 16MB

I have noticed the same. I think in the "real" world your compaction throughput is limited
by other things. If I had to speculate I would say that compaction can remove expired tombstones,
however doing this requires bloom filter checks, etc.

I think that setting is more important with multi threaded compaction and/or more compaction
slots. In those cases it may actually throttle something.


On Wed, Apr 24, 2013 at 3:54 PM, Hiller, Dean <Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov>>
wrote:
I was wondering about the compactionthroughput.  I never see ours get even close to 16MB
and I thought this is supposed to throttle compaction, right?  Ours is constantly less than
3MB/sec from looking at our logs or do I have this totally wrong?  How can I see the real
throughput so that I can understand how to throttle it when I need to?

94,940,780 bytes to 95,346,024 (~100% of original) in 38,438ms = 2.365603MB/s.  2,350,114
total rows, 2,350,022 unique.  Row merge counts were {1:2349930, 2:92, }

Thanks,
Dean
Mime
View raw message