Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7DBE510198 for ; Wed, 18 Sep 2013 19:39:11 +0000 (UTC) Received: (qmail 96439 invoked by uid 500); 18 Sep 2013 19:39:02 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 96416 invoked by uid 500); 18 Sep 2013 19:39:01 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 96380 invoked by uid 99); 18 Sep 2013 19:38:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2013 19:38:59 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [207.46.163.154] (HELO na01-bl2-obe.outbound.protection.outlook.com) (207.46.163.154) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2013 19:38:54 +0000 Received: from BN1PR07CA034.namprd07.prod.outlook.com (10.141.56.49) by BN1PR07MB008.namprd07.prod.outlook.com (10.255.225.38) with Microsoft SMTP Server (TLS) id 15.0.775.9; Wed, 18 Sep 2013 19:38:12 +0000 Received: from BN1AFFO11FD009.protection.gbl (2a01:111:f400:7c10::188) by BN1PR07CA034.outlook.office365.com (2a01:111:e400:2a::49) with Microsoft SMTP Server (TLS) id 15.0.775.9 via Frontend Transport; Wed, 18 Sep 2013 19:38:12 +0000 Received: from xedge2.nrel.gov (192.174.58.243) by BN1AFFO11FD009.mail.protection.outlook.com (10.58.52.69) with Microsoft SMTP Server (TLS) id 15.0.775.5 via Frontend Transport; Wed, 18 Sep 2013 19:38:12 +0000 Received: from XHUBA.nrel.gov (10.20.4.58) by xedge2.nrel.gov (192.174.58.243) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 18 Sep 2013 13:38:09 -0600 Received: from MAILBOX2.nrel.gov ([fe80::48b0:b121:8465:5e5]) by XHUBA.nrel.gov ([::1]) with mapi; Wed, 18 Sep 2013 13:38:10 -0600 From: "Hiller, Dean" To: "user@cassandra.apache.org" Date: Wed, 18 Sep 2013 13:38:07 -0600 Subject: Re: What is the ideal value for sstable_size_in_mb when using LeveledCompactionStrategy ? Thread-Topic: What is the ideal value for sstable_size_in_mb when using LeveledCompactionStrategy ? Thread-Index: Ac60ppspWaVMa5XURXm796UNjlzRUg== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.7.130812 acceptlanguage: en-US Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Forefront-Antispam-Report: CIP:192.174.58.243;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(189002)(199002)(51914003)(24454002)(377454003)(19580405001)(44976005)(6806004)(83322001)(69226001)(81542001)(81342001)(4396001)(50986001)(47976001)(47736001)(74662001)(53806001)(47446002)(74502001)(16796002)(31966008)(49866001)(46102001)(83506001)(51856001)(36756003)(74876001)(59766001)(54356001)(79102001)(53416003)(50466002)(76786001)(83072001)(76796001)(77096001)(77982001)(76176001)(56816003)(74706001)(81816001)(76482001)(65816001)(80976001)(80022001)(19580395003)(74366001)(81686001)(56776001)(20776003)(47776003)(54316002)(23746002)(63696002)(24704002);DIR:OUT;SFP:;SCL:1;SRVR:BN1PR07MB008;H:xedge2.nrel.gov;CLIP:192.174.58.243;FPR:;RD:xedge2.nrel.gov;MX:1;A:1;LANG:en; X-Forefront-PRVS: 09730BD177 X-OriginatorOrg: nrel.gov X-Virus-Checked: Checked by ClamAV on apache.org Sorry, bad bad typo=85..300G is what I meant. Cassandra heavily advises to stay under 1T per node or you run into big tro= ubles and most people stay under 500G per node. Later, Dean From: Jayadev Jayaraman > Reply-To: "user@cassandra.apache.org" > Date: Wednesday, September 18, 2013 1:30 PM To: "user@cassandra.apache.org" > Subject: Re: What is the ideal value for sstable_size_in_mb when using Leve= ledCompactionStrategy ? Thanks for the quick reply. We've already upped the ulimit as high as our L= inux distro allows us to ( around 1.8 million ). I have a follow-up question. I see that the size of individual nodes in you= r use case is quite massive. Does the safe number vary widely based on diff= erences in underlying hardware, or would you say from experience that somet= hing around 50M for medium to large datasets ( with upped file-descriptor l= imits ) is safe for most medium-sized (1 - 5 TB per node) to high-end (hund= reds of TB) hardware ? On Wed, Sep 18, 2013 at 3:15 PM, Hiller, Dean > wrote: 1. Always in cassandra up your file descriptor limits on linux and even i= n 0.7 that was the recommendation so cassandra could open tons of files 2. We use 50M for our LCS with no performance issues. We had it 10M on o= ur previous with no issues but a huge amount of files of course with our 30= 0T per node. Dean From: Jayadev Jayaraman >> Reply-To: "user@cassandra.apache.org>" >> Date: Wednesday, September 18, 2013 1:02 PM To: "user@cassandra.apache.org>" >> Subject: What is the ideal value for sstable_size_in_mb when using LeveledC= ompactionStrategy ? We have set up a 24 node (m1.xlarge nodes, 1.7 TB per node) cassandra clust= er on Amazon EC2 : version=3D1.2.9 replication factor =3D 2 snitch=3DEC2Snitch placement_strategy=3DNetworkTopologyStrategy (with 12 nodes each in 2 avail= ability zones) Background on our use-case : We plan on using hadoop with sstableloader to load 10GB+ of analytics data = per day ( 100 million+ row keys, 5 or so columns per day on average.) . We = have chosen LeveledCompactionStrategy in the hope that it constrains the nu= mber of SSTables that are read in order to retrieve a sliced-predicate for = a row. We don't want too many file-sockets ( > 1000) open to SSTables by th= e Cassandra JVM as this has caused us network / unreachability issues befor= e. We faced this when we were on cassandra 0.8.9 and we were using SizeTier= edCompactionStrategy and in order to mitigate this, we ran minor compaction= daily and major compaction semi-regularly to ensure as few SSTable files a= s possible on disk. If we use LeveledCompactionStrategy with a small value for sstable_size_in_= mb ( default =3D 5 MB ) , wouldn't that result in a very large number of SS= Table files on disk ? How does that affect the number of file-sockets open = (reading the docs, I get the impression that the number of SSTable seeks pe= r query is reduced by a large margin) ? But if we use a larger value for ss= table_size_in_mb, say around 200 MB, there will be 800 MB of small uncompac= ted SSTables on disk per column-family to which there will inevitably be fi= le-sockets open. All in all, can someone help us figure out what we should set the value of = sstable_size_in_mb to ? I figure it's not a very good idea to set it to a l= arger value but I don't know how things perform if we set it to a small val= ue. Do we have to run major compaction regularly in this case too ? Thanks Jayadev