Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of tholzer@wetafx.co.nz
 designates 110.232.144.26 as permitted sender)
Message-ID: <4E499E85.9060208@wetafx.co.nz>
Date: Tue, 16 Aug 2011 10:32:37 +1200
From: Teijo Holzer <tholzer@wetafx.co.nz>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:5.0) Gecko/20110705 Thunderbird/5.0
MIME-Version: 1.0
To: user@cassandra.apache.org
CC: Philippe <watcherfr@gmail.com>
Subject: Re: Scalability question
References: 
 <CAHwsXY=PY5Q_Qhc4Vrj_DtQCn3WAnriih9tcvwq3gqPFtuVzvw@mail.gmail.com>
 <78F2E2FF-B24D-44BB-BC51-F07E5B8CA03B@thelastpickle.com>
 <CAHwsXYnffRc-eQOUf8YL+3BZvVkSkJoEBPu=V4Di6LxWAt7WLw@mail.gmail.com>
 <CAHwsXY=8J0fAQ-_8yBD_7mdSMiK2mx90rBGazKO=dC0rae5bZg@mail.gmail.com>
In-Reply-To: 
 <CAHwsXY=8J0fAQ-_8yBD_7mdSMiK2mx90rBGazKO=dC0rae5bZg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

we have come across this as well. We run continuously run rolling repairs 
followed by major compactions followed by a gc() (or node restart) to get rid 
of all these sstables files. Combined with aggressive ttls on most inserts, the 
cluster stays nice and lean.

You don't want your working set to grow indefinitely.

Cheers,

	T.


On 16/08/11 08:08, Philippe wrote:
> Forgot to mention that stopping & restarting the server brought the data
> directory down to 283GB in less than 1 minute.
>
> Philippe
> 2011/8/15 Philippe <watcherfr@gmail.com <mailto:watcherfr@gmail.com>>
>
>         It's another reason to avoid major / manual compactions which create a
>         single big SSTable. Minor compactions keep things in buckets   which
>         means newer SSTable can be compacted needing to read the bigger older
>         tables.
>
>     I've never run a major/manual compaction on this ring.
>     In my case running repair on a "big" keyspace results in SSTables piling
>     up. My problematic node just filled up 483GB (yes, GB) of SSTTables. Here
>     are the biggest
>     ls -laSrh
>     (...)
>
>     -rw-r--r-- 1 cassandra cassandra  2.7G 2011-08-15 14:13
>     PUBLIC_MONTHLY_20-g-4581-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  2.7G 2011-08-15 14:52
>     PUBLIC_MONTHLY_20-g-4641-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  2.8G 2011-08-15 14:39
>     PUBLIC_MONTHLY_20-tmp-g-4878-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  2.9G 2011-08-15 15:00
>     PUBLIC_MONTHLY_20-g-4656-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  3.0G 2011-08-15 14:17
>     PUBLIC_MONTHLY_20-g-4599-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  3.0G 2011-08-15 15:11
>     PUBLIC_MONTHLY_20-g-4675-Data.db
>
>     -rw-r--r-- 3 cassandra cassandra  3.1G 2011-08-13 10:34
>     PUBLIC_MONTHLY_18-g-3861-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  3.2G 2011-08-15 14:41
>     PUBLIC_MONTHLY_20-tmp-g-4884-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  3.6G 2011-08-15 14:44
>     PUBLIC_MONTHLY_20-tmp-g-4894-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  3.8G 2011-08-15 14:56
>     PUBLIC_MONTHLY_20-tmp-g-4934-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  3.8G 2011-08-15 14:46
>     PUBLIC_MONTHLY_20-tmp-g-4905-Data.db
>
>     -rw-r--r-- 1 cassandra cassandra  4.0G 2011-08-15 14:57
>     PUBLIC_MONTHLY_20-tmp-g-4935-Data.db
>
>     -rw-r--r-- 3 cassandra cassandra  5.9G 2011-08-13 12:53
>     PUBLIC_MONTHLY_19-g-4219-Data.db
>
>     -rw-r--r-- 3 cassandra cassandra  6.0G 2011-08-13 13:57
>     PUBLIC_MONTHLY_20-g-4538-Data.db
>
>     -rw-r--r-- 3 cassandra cassandra   12G 2011-08-13 09:27
>     PUBLIC_MONTHLY_20-g-4501-Data.db
>
>
>     On the other nodes the same directory is around 69GB. Why are there so
>     fewer large files there and so many big ones on the repairing node ?
>       -rw-r--r-- 1 cassandra cassandra 434M 2011-08-15 16:02
>     PUBLIC_MONTHLY_17-g-3525-Data.db
>     -rw-r--r-- 1 cassandra cassandra 456M 2011-08-15 15:50
>     PUBLIC_MONTHLY_19-g-4253-Data.db
>     -rw-r--r-- 1 cassandra cassandra 485M 2011-08-15 14:30
>     PUBLIC_MONTHLY_20-g-5280-Data.db
>     -rw-r--r-- 1 cassandra cassandra 572M 2011-08-15 15:15
>     PUBLIC_MONTHLY_18-g-3774-Data.db
>     -rw-r--r-- 2 cassandra cassandra 664M 2011-08-09 15:39
>     PUBLIC_MONTHLY_20-g-4893-Index.db
>     -rw-r--r-- 2 cassandra cassandra 811M 2011-08-11 21:27
>     PUBLIC_MONTHLY_16-g-2597-Data.db
>     -rw-r--r-- 2 cassandra cassandra 915M 2011-08-13 04:00
>     PUBLIC_MONTHLY_18-g-3695-Data.db
>     -rw-r--r-- 1 cassandra cassandra 925M 2011-08-15 03:39
>     PUBLIC_MONTHLY_17-g-3454-Data.db
>     -rw-r--r-- 1 cassandra cassandra 1.3G 2011-08-15 13:46
>     PUBLIC_MONTHLY_19-g-4199-Data.db
>     -rw-r--r-- 2 cassandra cassandra 1.5G 2011-08-10 15:37
>     PUBLIC_MONTHLY_17-g-3218-Data.db
>     -rw-r--r-- 1 cassandra cassandra 1.9G 2011-08-15 14:35
>     PUBLIC_MONTHLY_20-g-5281-Data.db
>     -rw-r--r-- 2 cassandra cassandra 2.1G 2011-08-10 16:33
>     PUBLIC_MONTHLY_19-g-3946-Data.db
>     -rw-r--r-- 2 cassandra cassandra 3.1G 2011-08-10 22:23
>     PUBLIC_MONTHLY_18-g-3509-Data.db
>     -rw-r--r-- 2 cassandra cassandra 4.0G 2011-08-10 18:18
>     PUBLIC_MONTHLY_20-g-5024-Data.db
>     -rw------- 2 cassandra cassandra 5.1G 2011-08-09 15:23
>     PUBLIC_MONTHLY_19-g-3847-Data.db
>     -rw-r--r-- 2 cassandra cassandra 9.6G 2011-08-09 15:39
>     PUBLIC_MONTHLY_20-g-4893-Data.db
>
>     This whole compaction thing is getting me worried : how are sites in
>     production dealing with SSTables becoming larger and larger and thus taking
>     longer and longer to compact ? Adding nodes every couple of weeks ?
>
>     Philippe
>
>