Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <1331849116.29548.135.camel@cph-drift2.eur.adobe.com>
References: <1331849116.29548.135.camel@cph-drift2.eur.adobe.com>
Date: Thu, 15 Mar 2012 22:39:39 +0000
Message-ID: 
 <CADJL=w5vhwD6iQ3KiPCrRnZK_VbUwU64LqqrrXRWba9OzrfPMw@mail.gmail.com>
Subject: Re: 1.0.8 with Leveled compaction - Possible issues
From: Thomas van Neerijnen <tom@bossastudios.com>
To: user@cassandra.apache.org, jelmerfj@adobe.com
Content-Type: multipart/alternative; boundary=0015174781fe7842e504bb4fc488

--0015174781fe7842e504bb4fc488
Content-Type: text/plain; charset=ISO-8859-1

Heya

I'd suggest staying away from Leveled Compaction until 1.0.9.
For the why see this great explanation I got from Maki Watanabe on the
list:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201203.mbox/%3CCALqbeQbQ=d-hORVhA-LHOo_a5j46fQrsZMm+OQgfkgR=4RRQJQ@mail.gmail.com%3E
Keep an eye on that one because I'm busy testing one of his suggestions,
I'll post back with the results soon.

My understanding is after a change in compaction or compression, until you
run an upgradesstables on all the nodes the current sstables will have the
old schema settings, only new ones get the new format. Obviously this
compounds the issue I mentioned above tho.
Be warned, an upgradesstables can take a long time so maybe keep an eye on
the number of files around vs over 5MB to get an idea of progress. Maybe
someone else knows a better way?

You can change back and forth between compression and compaction options
quite safely, but again you need an upgradesstables to remove it from
current sstables.

In my experience I've safely applied compression and leveled compaction to
the same CF at the same time without issue so I guess it's ok:)

On Thu, Mar 15, 2012 at 10:05 PM, Johan Elmerfjord <jelmerfj@adobe.com>wrote:

> **
> Hi, I'm testing the community-version of Cassandra 1.0.8.
> We are currently on 0.8.7 in our production-setup.
>
> We have 3 Column Families that each takes between 20 and 35 GB on disk per
> node. (8*2 nodes total)
> We would like to change to Leveled Compaction - and even try compression
> as well to reduce the space needed for compactions.
> We are running on SSD-drives as latency is a key-issue.
>
> As test I have imported one Column Family from 3 production-nodes to a 3
> node test-cluster.
> The data on the 3 nodes ranges from 19-33GB. (with at least one large
> SSTable (Tiered size - recently compacted)).
>
> After loading this data to the 3 test-nodes, and running scrub and repair,
> I took a backup of the data so I have good test-set of data to work on.
> Then I changed changed to leveled compaction, using the cassandra-cli:
>
> UPDATE COLUMN FAMILY TestCF1 WITH
> compaction_strategy=LeveledCompactionStrategy;
> I could see the change being written to the logfile on all nodes.
>
> Then I don't know for for sure if I need to run anything else to make the
> change happen - or if it's just to wait.
> My test-cluster does not receive new data.
>
> For this  KS & CF and on each of the nodes I have tried some or several
> of: upgradesstable, scrub, compact, cleanup and repair - each task taking
> between 40 minutes and 4 hours.
> With the exception of compact that returns almost immediately with no
> visible compactions made.
>
> On some node I ended up with over 30000 files with the default 5MB size
> for leveled compaction, on another node it didn't look like anything has
> been done and I still have a 19GB SSTable.
>
> I then made another change.
> UPDATE COLUMN FAMILY TestCF1 WITH
> compaction_strategy=LeveledCompactionStrategy AND
> compaction_strategy_options=[{sstable_size_in_mb: 64}];
> WARNING: [{}] strategy_options syntax is deprecated, please use {}
> Which is probably wrong in the documentation - and should be:
> UPDATE COLUMN FAMILY TestCF1 WITH
> compaction_strategy=LeveledCompactionStrategy AND
> compaction_strategy_options={sstable_size_in_mb: 64};
>
> I think that we will be able to find the data in 3 searches with a 64MB
> size - and still only use around 700MB while doing compactions - and keep
> the number of files ~3000 per CF.
>
> A few days later it looks like I still have a mix between original huge
> SStables, 5MB once - and some nodes has 64MB files as well.
> Do I need to do something special to clean this up?
> I have tried another scrub /upgradesstables/clean - but nothing seems to
> do any change to me.
>
> Finally I have also tried to enable compression:
> UPDATE COLUMN FAMILY TestCF1 WITH
> compression_options=[{sstable_compression:SnappyCompressor,
> chunk_length_kb:64}];
> - which results in the same [{}] - warning.
>
> As you can see below - this created CompressionInfo.db - files on some
> nodes - but not on all.
>
> *Is there a way I can force Teired sstables to be converted into Leveled
> once - and then to compression as well?*
> *Why are the original file (Tiered Sized SSTables still present on
> testnode1 - when is it supposed to delete them?*
>
> *Can I change back and forth between compression (on/off - or chunksizes)
> - and between Leveled vs Size Tiered compaction?*
> *Is there a way to see if the node is done - or waiting for something?*
> *When is it safe to apply another setting - does it have to complete one
> reorg before moving on to the next?*
>
> *Any input or own experiences are warmly welcome.*
>
> Best regards, Johan
>
>
> Some lines of example directory-listings below.:
>
> Some files for testnode 3. (looks like it's still have the original Size
> Tiered files around, and a mixture of compressed 64MB files - and 5MB
> files?
>
> total 19G
> drwxr-xr-x 3 cass cass 4.0K Mar 13 17:11 snapshots
> -rw-r--r-- 1 cass cass 6.0G Mar 13 18:42 TestCF1-hc-6346-Index.db
> -rw-r--r-- 1 cass cass 1.3M Mar 13 18:42 TestCF1-hc-6346-Filter.db
> -rw-r--r-- 1 cass cass  13G Mar 13 18:42 TestCF1-hc-6346-Data.db
> -rw-r--r-- 1 cass cass 2.4M Mar 13 18:42 TestCF1-hc-6346-CompressionInfo.db
> -rw-r--r-- 1 cass cass 4.3K Mar 13 18:42 TestCF1-hc-6346-Statistics.db
> -rw-r--r-- 1 cass cass 195K Mar 13 18:42 TestCF1-hc-6347-Filter.db
> -rw-r--r-- 1 cass cass 4.9M Mar 13 18:42 TestCF1-hc-6347-Index.db
> -rw-r--r-- 1 cass cass 9.0M Mar 13 18:42 TestCF1-hc-6347-Data.db
> -rw-r--r-- 1 cass cass 4.3K Mar 13 18:42 TestCF1-hc-6347-Statistics.db
> -rw-r--r-- 1 cass cass 2.0K Mar 13 18:42 TestCF1-hc-6347-CompressionInfo.db
> -rw-r--r-- 1 cass cass  11K Mar 13 18:43 TestCF1-hc-6351-CompressionInfo.db
> -rw-r--r-- 1 cass cass  52M Mar 13 18:43 TestCF1-hc-6351-Data.db
> -rw-r--r-- 1 cass cass 1.1M Mar 13 18:43 TestCF1-hc-6351-Filter.db
> -rw-r--r-- 1 cass cass  28M Mar 13 18:43 TestCF1-hc-6351-Index.db
> -rw-r--r-- 1 cass cass 4.3K Mar 13 18:43 TestCF1-hc-6351-Statistics.db
> -rw-r--r-- 1 cass cass  401 Mar 13 18:43 TestCF1.json
> -rw-r--r-- 1 cass cass  950 Mar 13 18:43 TestCF1-hc-6350-CompressionInfo.db
> -rw-r--r-- 1 cass cass 4.3M Mar 13 18:43 TestCF1-hc-6350-Data.db
> -rw-r--r-- 1 cass cass  93K Mar 13 18:43 TestCF1-hc-6350-Filter.db
> -rw-r--r-- 1 cass cass 2.3M Mar 13 18:43 TestCF1-hc-6350-Index.db
> -rw-r--r-- 1 cass cass 4.3K Mar 13 18:43 TestCF1-hc-6350-Statistics.db
> -rw-r--r-- 1 cass cass  400 Mar 13 18:43 TestCF1-old.json
>
>
>
> Some DB-files ordered by size for testnode1:
> No compressed files - but has the original Size-tiered files - as well as
> 5MB Leveled compaction-files - but no 64 MB once.
>
> total 83G
> -rw-r--r-- 1 cass cass   33G Mar 13 07:14 TestCF1-hc-33504-Data.db
> -rw-r--r-- 1 cass cass   11G Mar 13 07:14 TestCF1-hc-33504-Index.db
> -rw-r--r-- 1 cass cass  407M Mar 13 07:14 TestCF1-hc-33504-Filter.db
> -rw-r--r-- 1 cass cass  5.1M Mar 13 05:27 TestCF1-hc-33338-Data.db
> -rw-r--r-- 1 cass cass  5.1M Mar 13 08:54 TestCF1-hc-38997-Data.db
> -rw-r--r-- 1 cass cass  5.1M Mar 13 07:15 TestCF1-hc-33513-Data.db
>
>
>
>   --
>
> *Johan Elmerfjord* | Sr. Systems Administration/Mgr, EMEA | Adobe
> Systems, Product Technical Operations | p. +45 3231 6008 | x86008 | cell. +46
> 735 101 444 | Jelmerfj@adobe.com
>
>

--0015174781fe7842e504bb4fc488
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Heya<br><br>I&#39;d suggest staying away from Leveled Compaction until 1.0.=
9.<br>For the why see this great explanation I got from Maki Watanabe on th=
e list: <a href=3D"http://mail-archives.apache.org/mod_mbox/cassandra-user/=
201203.mbox/%3CCALqbeQbQ=3Dd-hORVhA-LHOo_a5j46fQrsZMm+OQgfkgR=3D4RRQJQ@mail=
.gmail.com%3E">http://mail-archives.apache.org/mod_mbox/cassandra-user/2012=
03.mbox/%3CCALqbeQbQ=3Dd-hORVhA-LHOo_a5j46fQrsZMm+OQgfkgR=3D4RRQJQ@mail.gma=
il.com%3E</a><br>
Keep an eye on that one because I&#39;m busy testing one of his suggestions=
, I&#39;ll post back with the results soon.<br><br>My understanding is afte=
r a change in compaction or compression, until you run an upgradesstables o=
n all the nodes the current sstables will have the old schema settings, onl=
y new ones get the new format. Obviously this compounds the issue I mention=
ed above tho.<br>
Be warned, an upgradesstables can take a long time so maybe keep an eye on =
the number of files around vs over 5MB to get an idea of progress. Maybe so=
meone else knows a better way?<br><br>You can change back and forth between=
 compression and compaction options quite safely, but again you need an upg=
radesstables to remove it from current sstables.<br>
<br>In my experience I&#39;ve safely applied compression and leveled compac=
tion to the same CF at the same time without issue so I guess it&#39;s ok:)=
<br><br><div class=3D"gmail_quote">On Thu, Mar 15, 2012 at 10:05 PM, Johan =
Elmerfjord <span dir=3D"ltr">&lt;<a href=3D"mailto:jelmerfj@adobe.com">jelm=
erfj@adobe.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><u></u>


 =20
 =20

<div>
Hi, I&#39;m testing the community-version of Cassandra 1.0.8.<br>
We are currently on 0.8.7 in our production-setup.<br>
<br>
We have 3 Column Families that each takes between 20 and 35 GB on disk per =
node. (8*2 nodes total)<br>
We would like to change to Leveled Compaction - and even try compression as=
 well to reduce the space needed for compactions.<br>
We are running on SSD-drives as latency is a key-issue.<br>
<br>
As test I have imported one Column Family from 3 production-nodes to a 3 no=
de test-cluster.<br>
The data on the 3 nodes ranges from 19-33GB. (with at least one large SSTab=
le (Tiered size - recently compacted)).<br>
<br>
After loading this data to the 3 test-nodes, and running scrub and repair, =
I took a backup of the data so I have good test-set of data to work on.<br>
Then I changed changed to leveled compaction, using the cassandra-cli:<br>
<br>
UPDATE COLUMN FAMILY TestCF1 WITH compaction_strategy=3DLeveledCompactionSt=
rategy;<br>
I could see the change being written to the logfile on all nodes.<br>
<br>
Then I don&#39;t know for for sure if I need to run anything else to make t=
he change happen - or if it&#39;s just to wait.<br>
My test-cluster does not receive new data.<br>
 <br>
For this=A0 KS &amp; CF and on each of the nodes I have tried some or sever=
al of: upgradesstable, scrub, compact, cleanup and repair - each task takin=
g between 40 minutes and 4 hours.<br>
With the exception of compact that returns almost immediately with no visib=
le compactions made.<br>
<br>
On some node I ended up with over 30000 files with the default 5MB size for=
 leveled compaction, on another node it didn&#39;t look like anything has b=
een done and I still have a 19GB SSTable.<br>
<br>
I then made another change.<br>
UPDATE COLUMN FAMILY TestCF1 WITH compaction_strategy=3DLeveledCompactionSt=
rategy AND compaction_strategy_options=3D[{sstable_size_in_mb: 64}];<br>
WARNING: [{}] strategy_options syntax is deprecated, please use {}<br>
Which is probably wrong in the documentation - and should be:<br>
UPDATE COLUMN FAMILY TestCF1 WITH compaction_strategy=3DLeveledCompactionSt=
rategy AND compaction_strategy_options=3D{sstable_size_in_mb: 64};<br>
<br>
I think that we will be able to find the data in 3 searches with a 64MB siz=
e - and still only use around 700MB while doing compactions - and keep the =
number of files ~3000 per CF.<br>
<br>
A few days later it looks like I still have a mix between original huge SSt=
ables, 5MB once - and some nodes has 64MB files as well.<br>
Do I need to do something special to clean this up?<br>
I have tried another scrub /upgradesstables/clean - but nothing seems to do=
 any change to me.<br>
<br>
Finally I have also tried to enable compression:<br>
UPDATE COLUMN FAMILY TestCF1 WITH compression_options=3D[{sstable_compressi=
on:SnappyCompressor, chunk_length_kb:64}];<br>
- which results in the same [{}] - warning.<br>
<br>
As you can see below - this created CompressionInfo.db - files on some node=
s - but not on all.<br>
<br>
<b>Is there a way I can force Teired sstables to be converted into Leveled =
once - and then to compression as well?</b><br>
<b>Why are the original file (Tiered Sized SSTables still present on testno=
de1 - when is it supposed to delete them?</b><br>
<br>
<b>Can I change back and forth between compression (on/off - or chunksizes)=
 - and between Leveled vs Size Tiered compaction?</b><br>
<b>Is there a way to see if the node is done - or waiting for something?</b=
><br>
<b>When is it safe to apply another setting - does it have to complete one =
reorg before moving on to the next?</b><br>
<br>
<b>Any input or own experiences are warmly welcome.</b><br>
<br>
Best regards, Johan<br>
<br>
<br>
Some lines of example directory-listings below.:<br>
<br>
Some files for testnode 3. (looks like it&#39;s still have the original Siz=
e Tiered files around, and a mixture of compressed 64MB files - and 5MB fil=
es?
<pre>total 19G
drwxr-xr-x 3 cass cass 4.0K Mar 13 17:11 snapshots
-rw-r--r-- 1 cass cass 6.0G Mar 13 18:42 TestCF1-hc-6346-Index.db
-rw-r--r-- 1 cass cass 1.3M Mar 13 18:42 TestCF1-hc-6346-Filter.db
-rw-r--r-- 1 cass cass=A0 13G Mar 13 18:42 TestCF1-hc-6346-Data.db
-rw-r--r-- 1 cass cass 2.4M Mar 13 18:42 TestCF1-hc-6346-CompressionInfo.db
-rw-r--r-- 1 cass cass 4.3K Mar 13 18:42 TestCF1-hc-6346-Statistics.db
-rw-r--r-- 1 cass cass 195K Mar 13 18:42 TestCF1-hc-6347-Filter.db
-rw-r--r-- 1 cass cass 4.9M Mar 13 18:42 TestCF1-hc-6347-Index.db
-rw-r--r-- 1 cass cass 9.0M Mar 13 18:42 TestCF1-hc-6347-Data.db
-rw-r--r-- 1 cass cass 4.3K Mar 13 18:42 TestCF1-hc-6347-Statistics.db
-rw-r--r-- 1 cass cass 2.0K Mar 13 18:42 TestCF1-hc-6347-CompressionInfo.db
-rw-r--r-- 1 cass cass=A0 11K Mar 13 18:43 TestCF1-hc-6351-CompressionInfo.=
db
-rw-r--r-- 1 cass cass=A0 52M Mar 13 18:43 TestCF1-hc-6351-Data.db
-rw-r--r-- 1 cass cass 1.1M Mar 13 18:43 TestCF1-hc-6351-Filter.db
-rw-r--r-- 1 cass cass=A0 28M Mar 13 18:43 TestCF1-hc-6351-Index.db
-rw-r--r-- 1 cass cass 4.3K Mar 13 18:43 TestCF1-hc-6351-Statistics.db
-rw-r--r-- 1 cass cass=A0 401 Mar 13 18:43 TestCF1.json
-rw-r--r-- 1 cass cass=A0 950 Mar 13 18:43 TestCF1-hc-6350-CompressionInfo.=
db
-rw-r--r-- 1 cass cass 4.3M Mar 13 18:43 TestCF1-hc-6350-Data.db
-rw-r--r-- 1 cass cass=A0 93K Mar 13 18:43 TestCF1-hc-6350-Filter.db
-rw-r--r-- 1 cass cass 2.3M Mar 13 18:43 TestCF1-hc-6350-Index.db
-rw-r--r-- 1 cass cass 4.3K Mar 13 18:43 TestCF1-hc-6350-Statistics.db
-rw-r--r-- 1 cass cass=A0 400 Mar 13 18:43 TestCF1-old.json

</pre>
<br>
Some DB-files ordered by size for testnode1:<br>
No compressed files - but has the original Size-tiered files - as well as 5=
MB Leveled compaction-files - but no 64 MB once.
<pre>total 83G
-rw-r--r-- 1 cass cass=A0=A0 33G Mar 13 07:14 TestCF1-hc-33504-Data.db
-rw-r--r-- 1 cass cass=A0=A0 11G Mar 13 07:14 TestCF1-hc-33504-Index.db
-rw-r--r-- 1 cass cass=A0 407M Mar 13 07:14 TestCF1-hc-33504-Filter.db
-rw-r--r-- 1 cass cass=A0 5.1M Mar 13 05:27 TestCF1-hc-33338-Data.db
-rw-r--r-- 1 cass cass=A0 5.1M Mar 13 08:54 TestCF1-hc-38997-Data.db
-rw-r--r-- 1 cass cass=A0 5.1M Mar 13 07:15 TestCF1-hc-33513-Data.db
</pre><span class=3D"HOEnZb"><font color=3D"#888888">
<br>
<br>
<table cellpadding=3D"0" cellspacing=3D"0" width=3D"100%">
<tbody><tr>
<td>
-- <br>
<font><font color=3D"#000080">=A0</font></font> <br>
<b><font>Johan Elmerfjord</font></b> <font size=3D"1">| Sr. Systems Adminis=
tration/Mgr, EMEA | Adobe Systems, Product Technical Operations | p. <a hre=
f=3D"tel:%2B45%203231%206008" value=3D"+4532316008" target=3D"_blank">+45 3=
231 6008</a> | x86008 | cell. <a href=3D"tel:%2B46%20735%20101%20444" value=
=3D"+46735101444" target=3D"_blank">+46 735 101 444</a> | <a href=3D"mailto=
:Jelmerfj@adobe.com" target=3D"_blank">Jelmerfj@adobe.com</a></font> <br>

<br>
</td>
</tr>
</tbody></table>
</font></span></div>

</blockquote></div><br>

--0015174781fe7842e504bb4fc488--