Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3115218CAD for ; Fri, 30 Oct 2015 12:05:02 +0000 (UTC) Received: (qmail 28796 invoked by uid 500); 30 Oct 2015 12:04:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 28758 invoked by uid 500); 30 Oct 2015 12:04:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 28748 invoked by uid 99); 30 Oct 2015 12:04:56 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Oct 2015 12:04:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D4D9118098B for ; Fri, 30 Oct 2015 12:04:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id hULj0Lh9CtwE for ; Fri, 30 Oct 2015 12:04:43 +0000 (UTC) Received: from setentaytres29.nsprimario.com (setentaytres31.nsprimario.com [188.93.73.31]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id D5C9A439B6 for ; Fri, 30 Oct 2015 12:04:42 +0000 (UTC) X-No-Relay: not in my network Received: from mail-wm0-f43.google.com (mail-wm0-f43.google.com [74.125.82.43]) by setentaytres29.nsprimario.com (Postfix) with ESMTPSA id D65FC10A0F19 for ; Fri, 30 Oct 2015 13:04:34 +0100 (CET) Received: by wmec75 with SMTP id c75so10407287wme.1 for ; Fri, 30 Oct 2015 05:04:34 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.28.92.209 with SMTP id q200mr3293155wmb.52.1446206674445; Fri, 30 Oct 2015 05:04:34 -0700 (PDT) Received: by 10.27.85.31 with HTTP; Fri, 30 Oct 2015 05:04:34 -0700 (PDT) In-Reply-To: References: Date: Fri, 30 Oct 2015 12:04:34 +0000 Message-ID: Subject: Re: Cassandra Data Model with Narrow partition From: Carlos Alonso To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a1146ffb41a2eb90523513b01 --001a1146ffb41a2eb90523513b01 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Chandra, Narrow partition is probably your best choice, but you need to bucket data somehow, otherwise your partitions will soon become unmanageable and you'll have problems reading them, both because the partitions will become very big and also because of the tombstones that your expired records will generate. In general having a partition that can grow indefinitely is a bad idea, so I'd advice you to use time based artificial bucketing to limit the maximum size of your partitions to be as close as possible to the recommendations. Also 120+ columns sounds like quite many, is there a way you can separate in different cfs or maybe use collections? I'd advice to do some benchmarking here: http://mrcalonso.com/benchmarking-cassandra-models/. This post is a bit outdated as nowadays you can use cassandra-stress with your own models, but the idea is the same. About compactions I'd use DTCS or LCS, but given that you will have a big amount of tombstones due to TTLs I'd never go with STCS. Hope it helps! Carlos Alonso | Software Engineer | @calonso On 30 October 2015 at 10:55, wrote: > Hi, > > > > Could you please suggest if Narrow partition is a good choice for the > below use case. > > > > 1) Write heavy event log table with 50m inserts per day with a peak > load of 20K transaction per sec. There aren=E2=80=99t any updates/deletes= to > records inserted. Records are inserted with a TTL of 60 days (retention > period) > > 2) The table has a single primary key which is a sequence number (27 > digits) generated by source application > > 3) There are only two access patterns used =E2=80=93 one by using th= e > sequence number & the other using sequence number + event date (range sca= ns > also possible) > > 4) My target data model in Cassandra is partitioned with sequence > number as the primary key + event date as clustering columns to enable > range scans on date. > > 5) The Table has close to 120+ columns and the average row size > comes close to 32K bytes > > 6) Reads are very very less and account to <5% while inserts can be > close to 95%. > > 7) From a functional standpoint, I do not see any other columns that > can be part of primary key to keep the partition reasonable (<100MB) > > > > Questions: > > 1) Is Narrow partition an ideal choice for the above use case. > > 2) Is artificial bucketing an alternate choice to make the partition > reasonable > > 3) We are using varint as the data type for sequence number which is > 27 digits long. Is DECIMAL data type ? > > 4) Any suggestions on performance impacts during compaction ? > > > > Regards, Chandra Sekar KR > > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) an= d > may contain proprietary, confidential or privileged information. If you a= re > not the intended recipient, you should not disseminate, distribute or cop= y > this e-mail. Please notify the sender immediately and destroy all copies = of > this message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability > for any damage caused by any virus transmitted by this email. > www.wipro.com > --001a1146ffb41a2eb90523513b01 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Chandra,

Narrow partition is probabl= y your best choice, but you need to bucket data somehow, otherwise your par= titions will soon become unmanageable and you'll have problems reading = them, both because the partitions will become very big and also because of = the tombstones that your expired records will generate.

In general having a partition that can grow indefinitely is a bad ide= a, so I'd advice you to use time based artificial bucketing to limit th= e maximum size of your partitions to be as close as possible to the recomme= ndations.

Also 120+ columns sounds like quite many= , is there a way you can separate in different cfs or maybe use collections= ? I'd advice to do some benchmarking here:=C2=A0http://mrcalonso.com/benchmarking-= cassandra-models/. This post is a bit outdated as nowadays you can use = cassandra-stress with your own models, but the idea is the same.
=
About compactions I'd use DTCS or LCS, but given that yo= u will have a big amount of tombstones due to TTLs I'd never go with ST= CS.

Hope it helps!

Carlos Alonso | Software En= gineer |=C2=A0@calonso

On 30 October 2015 at 10:55, <cha= ndrasekar.krc@wipro.com> wrote:

Hi,

=C2=A0

Could you please suggest if Narrow partition is a=C2= =A0 good choice for the below use case.

=C2=A0

1)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Write heavy event log table with 50m inserts per day w= ith a peak load of 20K transaction per sec. There aren=E2=80=99t any update= s/deletes to records inserted. Records are inserted with a TTL of 60 days (= retention period)

2)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 The table has a single primary key which is a sequence= number (27 digits) generated by source application

3)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 There are only two access patterns used =E2=80=93 one = by using the sequence number & the other using sequence number + event = date (range scans also possible)

4)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 My target data model in Cassandra is partitioned with = sequence number as the primary key + event date as clustering columns to en= able range scans on date.

5)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 The Table has close to 120+ columns and the average ro= w size comes close to 32K bytes

6)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Reads are very very less and account to <5% while i= nserts can be close to 95%.

7)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 From a functional standpoint, I do not see any other c= olumns that can be part of primary key to keep the partition reasonable (&l= t;100MB)

=C2=A0

Questions:

1)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Is Narrow partition an ideal choice for the above use = case.

2)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Is artificial bucketing an alternate choice to make th= e partition reasonable

3)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We are using varint as the data type for sequence numb= er which is 27 digits long. Is DECIMAL data type ?

4)= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Any suggestions on performance impacts during compacti= on ?

=C2=A0

Regards, Chandra Sekar KR

=C2=A0

The information contained in this electronic message and any attachments to= this message are intended for the exclusive use of the addressee(s) and ma= y contain proprietary, confidential or privileged information. If you are n= ot the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender = immediately and destroy all copies of this message and any attachments. WAR= NING: Computer viruses can be transmitted via email. The recipient should c= heck this email and any attachments for the presence of viruses. The company accepts no liability for any dama= ge caused by any virus transmitted by this email. www.wipro.com

--001a1146ffb41a2eb90523513b01--