Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2FCEB10062 for ; Mon, 17 Jun 2013 04:56:19 +0000 (UTC) Received: (qmail 91869 invoked by uid 500); 17 Jun 2013 04:56:16 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 91848 invoked by uid 500); 17 Jun 2013 04:56:16 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 91835 invoked by uid 99); 17 Jun 2013 04:56:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jun 2013 04:56:15 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: local policy) Received: from [74.125.149.19] (HELO na3sys009aog138.obsmtp.com) (74.125.149.19) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 17 Jun 2013 04:56:08 +0000 Received: from mail-vb0-f51.google.com ([209.85.212.51]) (using TLSv1) by na3sys009aob138.postini.com ([74.125.148.12]) with SMTP ID DSNKUb6WvTw2bE4tnBzT8MFgXDseMTxmMAfT@postini.com; Sun, 16 Jun 2013 21:55:48 PDT Received: by mail-vb0-f51.google.com with SMTP id x17so1647464vbf.10 for ; Sun, 16 Jun 2013 21:55:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=sj/VoMHOck250Yt9Ng4JDu4ABGfP8mUYnBPt4HaNaxo=; b=In6JqQHmhG9ijunkuufv+ZRWwwBKDVMlo1oRdblGmiqjVpps+pss9mGQ8Lk0scImo+ Myyk5pmWxwyRVWdW1x9Th1XyqcIKUh3BdaW1+fL77TiJM6m9GfntLd9t4lLQjy81eGbl Gxr7gv42hz527iHQ3ostiAe0aY8alQBqyBpP9W2CIHQnlA5jcNtL934MVoz4s8L2sePW zobValp5Hd++OFmM3tJYPMraGHP/mZVyjKXNVYY4FXHl9EMVRUz5gS5FkU+oh37xazb6 mbXxVaaKOGpgJ5iIcIPFnr4EwlFjDm58VYkz+Xif03oJoaRPoTm5ugT3gtuGhaCm6A1N lT7w== X-Received: by 10.52.249.41 with SMTP id yr9mr3125023vdc.17.1371444920549; Sun, 16 Jun 2013 21:55:20 -0700 (PDT) X-Received: by 10.52.249.41 with SMTP id yr9mr3125021vdc.17.1371444920446; Sun, 16 Jun 2013 21:55:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.181.9 with HTTP; Sun, 16 Jun 2013 21:54:59 -0700 (PDT) In-Reply-To: References: From: Franc Carter Date: Mon, 17 Jun 2013 14:54:59 +1000 Message-ID: Subject: Re: Large number of files for Leveled Compaction To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e01294a805008e104df52677a X-Gm-Message-State: ALoCoQmz+i8YdG05fZf9eJBzoUnJOxFWE2kGK7C97oM95WBXmOSGnwxUDHTOqXtNQzwMMT0JtJoM5EufiKA+6T//jWBP0eo8hLzfWQScvMqpcihJtyvqQ0RCMMPbYPQjjQMGKudtjNgQQ5VEMxMTnGbGwR48pF9BAeufSWMATnTtiTarAHQqtjo= X-Virus-Checked: Checked by ClamAV on apache.org --089e01294a805008e104df52677a Content-Type: text/plain; charset=ISO-8859-1 On Mon, Jun 17, 2013 at 2:47 PM, Manoj Mainali wrote: > With LeveledCompaction, each sstable size is fixed and is defined by > sstable_size_in_mb in the compaction configuration of CF definition and > default value is 5MB. In you case, you may have not defined your own value, > that is why your each sstable is 5MB. And if you dataset is huge, you will > see a lot of sstable counts. > Ok, seems like I do have (at least) an incomplete understanding. I realise that the minimum size is 5MB, but I thought compaction would merge these into a smaller number of larger sstables ? thanks > Cheers > > Manoj > > > On Fri, Jun 7, 2013 at 1:44 PM, Franc Carter wrote: > >> >> Hi, >> >> We are trialling Cassandra-1.2(.4) with Leveled compaction as it looks >> like it may be a win for us. >> >> The first step of testing was to push a fairly large slab of data into >> the Column Family - we did this much faster (> x100) than we would in a >> production environment. This has left the Column Family with about 140,000 >> files in the Column Family directory which seems way too high. On two of >> the nodes the CompactionStats show 2 outstanding tasks and on a third node >> there are over 13,000 outstanding tasks. However from looking at the log >> activity it looks like compaction has finished on all nodes. >> >> Is this number of files expected/normal ? >> >> cheers >> >> -- >> >> *Franc Carter* | Systems architect | Sirca Ltd >> >> >> franc.carter@sirca.org.au | www.sirca.org.au >> >> Tel: +61 2 8355 2514 >> >> Level 4, 55 Harrington St, The Rocks NSW 2000 >> >> PO Box H58, Australia Square, Sydney NSW 1215 >> >> >> > -- *Franc Carter* | Systems architect | Sirca Ltd franc.carter@sirca.org.au | www.sirca.org.au Tel: +61 2 8355 2514 Level 4, 55 Harrington St, The Rocks NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215 --089e01294a805008e104df52677a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Mon, Jun 17, 2013 at 2:47 PM, Manoj Mainali <mainalimanoj@gmail.co= m> wrote:
With LeveledCompaction, each sstable size is fixed and is = defined by sstable_size_in_mb in=A0the compaction configuration of CF defin= ition and default value is 5MB. In you case, you may have not defined your = own value, that is why your each sstable is 5MB. And if you dataset is huge= , you will see a lot of sstable counts.


Ok, seems like I do have (at least) an inco= mplete understanding. I realise that the minimum size is 5MB, but I thought= compaction would merge these into a smaller number of larger sstables ?
thanks


Cheers

Manoj


On Fri, Jun 7, = 2013 at 1:44 PM, Franc Carter <franc.carter@sirca.org.au> wrote:

Hi,

= We are trialling Cassandra-1.2(.4) with Leveled compaction as it looks like= it may be a win for us.

The first step of testing was to push a fairly large slab of data into the = Column Family - we did this much faster (> x100) than we would in a prod= uction environment. This has left the Column Family with about 140,000 file= s in the Column Family directory which seems way too high. On two of the no= des the CompactionStats show 2 outstanding tasks and on a third node there = are over 13,000 outstanding tasks. However from looking at the log activity= it looks like compaction has finished on all nodes.

Is this number of files expected/normal ?

cheers

--

Franc Carter<= /b> |<= /span> Systems architect | Sirca Ltd

franc.carter@sirca.org.au=A0|=A0www.sirca.org.au

Tel:= =A0+61 2 8355 2514

Level 4, 55 Harrington St, The Rocks NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215<= /span>






--

Franc Carter<= /b> |<= /span> Systems architect | Sirca Ltd

franc.carter@sirca.org.au=A0|=A0www.sirca.org.au

Tel:= =A0+61 2 8355 2514

Level 4, 55 Harrington St, The Rocks NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215<= /span>


--089e01294a805008e104df52677a--