Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 912B9ED28 for ; Sat, 16 Feb 2013 17:26:56 +0000 (UTC) Received: (qmail 94639 invoked by uid 500); 16 Feb 2013 17:26:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94604 invoked by uid 500); 16 Feb 2013 17:26:53 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94595 invoked by uid 99); 16 Feb 2013 17:26:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Feb 2013 17:26:53 +0000 X-ASF-Spam-Status: No, hits=2.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [98.138.91.163] (HELO nm3-vm4.bullet.mail.ne1.yahoo.com) (98.138.91.163) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Feb 2013 17:26:47 +0000 Received: from [98.138.226.179] by nm3.bullet.mail.ne1.yahoo.com with NNFMP; 16 Feb 2013 17:26:26 -0000 Received: from [98.138.226.128] by tm14.bullet.mail.ne1.yahoo.com with NNFMP; 16 Feb 2013 17:26:26 -0000 Received: from [127.0.0.1] by smtp215.mail.ne1.yahoo.com with NNFMP; 16 Feb 2013 17:26:26 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1361035586; bh=2xoPktqIlAshU0Qb8+/iZdAivoyK1uKMRTlIZc06ttY=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type; b=6cRNE+rIiZmIy6L8db7eyGx4kKx1Hs3eYbEosfeKwSvl5B+msgtsirqqP4ITzhXuJt6VvknQo9CD9iCqxPwJZS1E4GKVZQLCzT9GslzMOg+95Wgqub1g9Nc4CPK3a33fyEZDoVP4BoBxRLS2MCMQvViY1HAwJh2PYIFSjNC5KFo= X-Yahoo-Newman-Id: 583832.80106.bm@smtp215.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: Aq3yoW4VM1lqWP0f84hRycJAwOjCb7OG5HxQtSLNPUIVucN hVTVMKfbXHJwVgoTedPtpJMREfSrHvBuoVKyUvTMroIlY_0Eg8_8wByT0eII FupPsT_3M.XDF3zgShdMEMp_7nmftLXE_LvzH35QzoXJsueMN6HNVFqomMQF hhKz5UyurRrVh7XmKyYiGSIAcUQU9kSb8wIGzoIIQkQrqhJAnvxUrUJDqO3b fx82myiLxOA7PTKVJCTuXeif0S9ol0Wt.P.ykvyC6MyIJzlrQtP.7vD.b_Uu xa9ltVzx8n_RH4DQUdOt7s1EWj2kXFoB4AC0T3vj.gPdlXoIf3nkS3cqF7oq okVYAowb3ibv9dtLEK_ZhPgiEb3T1TD0LZGecQHG2Y59jOSYjfMAH9T141Jn 9WmhFh653naS4dNvcqq81.ico.0A86Q.f3TSBSJ5qZ95QnBGoIp_m.4F7KNx E5CxZWmeFeobPMqWAOFRxBT40hKwy__r2j.OMvaRDUNbJBupow1398uU7QRJ l7T781TnD5gdaKsAUkfSQtU7UT3WrbDhyRt9wCx9UUQM0JqztIkSq9zhH0uf TXp.8Iy5yqKAcbRmMoKf9yw-- X-Yahoo-SMTP: t0UN_U2swBCFgwLIRu70LU92TrvpdQ-- Received: from [192.168.1.5] (mtheroux2@76.118.248.45 with plain) by smtp215.mail.ne1.yahoo.com with SMTP; 16 Feb 2013 09:26:26 -0800 PST Message-ID: <511FC141.3050902@yahoo.com> Date: Sat, 16 Feb 2013 12:26:25 -0500 From: Mike User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: user@cassandra.apache.org Subject: Re: Size Tiered -> Leveled Compaction References: <511BB62D.2060807@yahoo.com> <1360875114.43272.YahooMailNeo@web160903.mail.bf1.yahoo.com> <87D968E5-56A0-4DA7-8676-BA90FF376761@yahoo.com> In-Reply-To: <87D968E5-56A0-4DA7-8676-BA90FF376761@yahoo.com> Content-Type: multipart/alternative; boundary="------------020401070200020104030200" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------020401070200020104030200 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Another piece of information that would be useful is advice on how to properly set the SSTable size for your usecase. I understand the default is 5MB, a lot of examples show the use of 10MB, and I've seen cases where people have set is as high as 200MB. Any information is appreciated, -Mike On 2/14/2013 4:10 PM, Michael Theroux wrote: > BTW, when I say "major compaction", I mean running the "nodetool > compact" command (which does a major compaction for Sized Tiered > Compaction). I didn't see the distribution of SSTables I expected > until I ran that command, in the steps I described below. > > -Mike > > On Feb 14, 2013, at 3:51 PM, Wei Zhu wrote: > >> I haven't tried to switch compaction strategy. We started with LCS. >> >> For us, after massive data imports (5000 w/seconds for 6 days), the >> first repair is painful since there is quite some data inconsistency. >> For 150G nodes, repair brought in about 30 G and created thousands of >> pending compactions. It took almost a day to clear those. Just be >> prepared LCS is really slow in 1.1.X. System performance degrades >> during that time since reads could go to more SSTable, we see 20 >> SSTable lookup for one read.. (We tried everything we can and >> couldn't speed it up. I think it's single threaded.... and it's not >> recommended to turn on multithread compaction. We even tried that, it >> didn't help )There is parallel LCS in 1.2 which is supposed to >> alleviate the pain. Haven't upgraded yet, hope it works:) >> >> http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 >> >> >> Since our cluster is not write intensive, only 100 w/seconds. I don't >> see any pending compactions during regular operation. >> >> One thing worth mentioning is the size of the SSTable, default is 5M >> which is kind of small for 200G (all in one CF) data set, and we are >> on SSD. It more than 150K files in one directory. (200G/5M = 40K >> SSTable and each SSTable creates 4 files on disk) You might want to >> watch that and decide the SSTable size. >> >> By the way, there is no concept of Major compaction for LCS. Just for >> fun, you can look at a file called $CFName.json in your data >> directory and it tells you the SSTable distribution among different >> levels. >> >> -Wei >> >> ------------------------------------------------------------------------ >> *From:* Charles Brophy > >> *To:* user@cassandra.apache.org >> *Sent:* Thursday, February 14, 2013 8:29 AM >> *Subject:* Re: Size Tiered -> Leveled Compaction >> >> I second these questions: we've been looking into changing some of >> our CFs to use leveled compaction as well. If anybody here has the >> wisdom to answer them it would be of wonderful help. >> >> Thanks >> Charles >> >> On Wed, Feb 13, 2013 at 7:50 AM, Mike > > wrote: >> >> Hello, >> >> I'm investigating the transition of some of our column families >> from Size Tiered -> Leveled Compaction. I believe we have some >> high-read-load column families that would benefit tremendously. >> >> I've stood up a test DB Node to investigate the transition. I >> successfully alter the column family, and I immediately noticed a >> large number (1000+) pending compaction tasks become available, >> but no compaction get executed. >> >> I tried running "nodetool sstableupgrade" on the column family, >> and the compaction tasks don't move. >> >> I also notice no changes to the size and distribution of the >> existing SSTables. >> >> I then run a major compaction on the column family. All pending >> compaction tasks get run, and the SSTables have a distribution >> that I would expect from LeveledCompaction (lots and lots of 10MB >> files). >> >> Couple of questions: >> >> 1) Is a major compaction required to transition from size-tiered >> to leveled compaction? >> 2) Are major compactions as much of a concern for >> LeveledCompaction as their are for Size Tiered? >> >> All the documentation I found concerning transitioning from Size >> Tiered to Level compaction discuss the alter table cql command, >> but I haven't found too much on what else needs to be done after >> the schema change. >> >> I did these tests with Cassandra 1.1.9. >> >> Thanks, >> -Mike >> >> >> >> > --------------020401070200020104030200 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Another piece of information that would be useful is advice on how to properly set the SSTable size for your usecase.  I understand the default is 5MB, a lot of examples show the use of 10MB, and I've seen cases where people have set is as high as 200MB.

Any information is appreciated,
-Mike

On 2/14/2013 4:10 PM, Michael Theroux wrote:
BTW, when I say "major compaction", I mean running the "nodetool compact" command (which does a major compaction for Sized Tiered Compaction).  I didn't see the distribution of SSTables I expected until I ran that command, in the steps I described below.  

-Mike

On Feb 14, 2013, at 3:51 PM, Wei Zhu wrote:

I haven't tried to switch compaction strategy. We started with LCS. 

For us, after massive data imports (5000 w/seconds for 6 days), the first repair is painful since there is quite some data inconsistency. For 150G nodes, repair brought in about 30 G and created thousands of pending compactions. It took almost a day to clear those. Just be prepared LCS is really slow in 1.1.X. System performance degrades during that time since reads could go to more SSTable, we see 20 SSTable lookup for one read.. (We tried everything we can and couldn't speed it up. I think it's single threaded.... and it's not recommended to turn on multithread compaction. We even tried that, it didn't help )There is parallel LCS in 1.2 which is supposed to alleviate the pain. Haven't upgraded yet, hope it works:)



Since our cluster is not write intensive, only 100 w/seconds. I don't see any pending compactions during regular operation. 

One thing worth mentioning is the size of the SSTable, default is 5M which is kind of small for 200G (all in one CF) data set, and we are on SSD.  It more than  150K files in one directory. (200G/5M = 40K SSTable and each SSTable creates 4 files on disk)  You might want to watch that and decide the SSTable size. 

By the way, there is no concept of Major compaction for LCS. Just for fun, you can look at a file called $CFName.json in your data directory and it tells you the SSTable distribution among different levels. 

-Wei

From: Charles Brophy <cbrophy@zulily.com>
To: user@cassandra.apache.org
Sent: Thursday, February 14, 2013 8:29 AM
Subject: Re: Size Tiered -> Leveled Compaction

I second these questions: we've been looking into changing some of our CFs to use leveled compaction as well. If anybody here has the wisdom to answer them it would be of wonderful help.

Thanks
Charles

On Wed, Feb 13, 2013 at 7:50 AM, Mike <mtheroux2@yahoo.com> wrote:
Hello,

I'm investigating the transition of some of our column families from Size Tiered -> Leveled Compaction.  I believe we have some high-read-load column families that would benefit tremendously.

I've stood up a test DB Node to investigate the transition.  I successfully alter the column family, and I immediately noticed a large number (1000+) pending compaction tasks become available, but no compaction get executed.

I tried running "nodetool sstableupgrade" on the column family, and the compaction tasks don't move.

I also notice no changes to the size and distribution of the existing SSTables.

I then run a major compaction on the column family.  All pending compaction tasks get run, and the SSTables have a distribution that I would expect from LeveledCompaction (lots and lots of 10MB files).

Couple of questions:

1) Is a major compaction required to transition from size-tiered to leveled compaction?
2) Are major compactions as much of a concern for LeveledCompaction as their are for Size Tiered?

All the documentation I found concerning transitioning from Size Tiered to Level compaction discuss the alter table cql command, but I haven't found too much on what else needs to be done after the schema change.

I did these tests with Cassandra 1.1.9.

Thanks,
-Mike





--------------020401070200020104030200--