incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: RF update
Date Thu, 18 Oct 2012 11:32:02 GMT
> Follow up question: Is it safe to abort the compactions happening after node repair?
It is always safe to abort a compaction. The purpose of compaction is to replicate the current
truth in a more compact format. It does not modify data, it just creates new files. The worse
case would be killing it between the time the new files are marked as non temp and the time
the old files are deleted. That would result in wasted disk space, but the truth in the system
would not change. 

> 
> > Question: These additional compactions seem redundant since there are no reads or
writes on the cluster after the first major compaction (immediately after the data load),
is that right?

Repair transfers a portion of the  -Data.db component from potentially multiple SSTables.
This may result in multiple new SStables being created on the receiving node. Once the files
are created they are processed in a similar way to when a memtable is flushed and so compaction
kicks in.

> And if so, what can we do to avoid them? We are currently waiting multiple days.

That fact that compaction is taking so long is odd. Have you checked the logs for GC problems?
if you are running an SSD backed instance and have turned off compaction throttling the high
IO throughput can result in mucho garbage. Faster is not always better. 

To improve your situation consider:

* disabling compaction by setting min_compaction_threshold and max_compaction_threshold to
0 via schema or nodetool
* disabling durable_writes to disable the commit log during the bulk load. 

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 17/10/2012, at 11:55 PM, Matthias Broecheler <me@matthiasb.com> wrote:

> Follow up question: Is it safe to abort the compactions happening after node repair?
> 
> On Mon, Oct 15, 2012 at 6:32 PM, Will Martin <will@voodoolunchbox.com> wrote:
> +1   It doesn't make sense that the xfr compactions are heavy unless they are translating
the file. This could be a protocol mismatch: however the requirements for node level compaction
and wire compaction I would expect to be pretty different.
> On Oct 15, 2012, at 4:42 PM, Matthias Broecheler wrote:
> 
> > Hey,
> >
> > we are writing a lot of data into a cassandra cluster for a batch loading use case.
We cannot use the sstable batch loader, so in order to speed up the loading process we are
using RF=1 while the data is loading. After the load is complete, we want to increase the
RF. For that, we are updating the RF in the schema and then run the node repair tool on each
cassandra instance to stream the data over. However, we are noticing that this process is
slowed down by a lot of compactions (the actually streaming of data only takes a couple of
minutes).
> >
> > Cassandra is already running a major compaction after the data loading process has
completed. But then, there are to be two more compactions (one on the sender and one on the
receiver) happening and those take a very long time even on the aws high i/o instance with
no compaction throttling.
> >
> > Question: These additional compactions seem redundant since there are no reads or
writes on the cluster after the first major compaction (immediately after the data load),
is that right? And if so, what can we do to avoid them? We are currently waiting multiple
days.
> >
> > Thank you very much for your help,
> > Matthias
> >
> 
> 
> 
> 
> -- 
> Matthias Broecheler, PhD
> http://www.matthiasb.com
> E-Mail: me@matthiasb.com


Mime
View raw message