Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E310E108D8 for ; Mon, 24 Feb 2014 16:52:09 +0000 (UTC) Received: (qmail 41102 invoked by uid 500); 24 Feb 2014 16:52:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 41070 invoked by uid 500); 24 Feb 2014 16:52:05 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 41061 invoked by uid 99); 24 Feb 2014 16:52:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Feb 2014 16:52:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rklaehn@gmail.com designates 209.85.214.180 as permitted sender) Received: from [209.85.214.180] (HELO mail-ob0-f180.google.com) (209.85.214.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Feb 2014 16:52:01 +0000 Received: by mail-ob0-f180.google.com with SMTP id vb8so7182487obc.11 for ; Mon, 24 Feb 2014 08:51:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wZx2FnkFTuDcUPvLy/w0wZsxEPS4Fsnsmq1a7OrB+9s=; b=Voloj06pbBStwGhHs9LkAeKjLa81FxWK2ECP+zwT+NxemzWEI+i/V4ZLvWukcjcvBp 3dVbu4cE/vTRXifjoYoALvhbwYY2ZCghIM1pT7uFroW/ZtrnxySbd75/PQERRsuEODNL OOZtSa1Y9/APQwwualOyDXDD+PHQvaeq2k1NBwA9W1DjfyIDYX9EbttmtyG2GCiUNFvk G4ROO9FaFSgDSNlMxttnnUhRfcEJm9TP8sQY28PxNm4ObY1SJ3QRQsDAdKFlFcu0wyLT RTp1htQ9rNMlh/GRaYjBs25bMt1Ot7Vd4SM/ySmbAtffQomAg1Dc97dcvWw67zi49V06 /tLg== MIME-Version: 1.0 X-Received: by 10.182.48.233 with SMTP id p9mr18298703obn.44.1393260700463; Mon, 24 Feb 2014 08:51:40 -0800 (PST) Received: by 10.76.168.197 with HTTP; Mon, 24 Feb 2014 08:51:40 -0800 (PST) In-Reply-To: References: Date: Mon, 24 Feb 2014 17:51:40 +0100 Message-ID: Subject: Re: Performance problem with large wide row inserts using CQL From: =?ISO-8859-1?Q?R=FCdiger_Klaehn?= To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b66f0bd220cef04f329c923 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b66f0bd220cef04f329c923 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Mon, Feb 24, 2014 at 11:47 AM, Sylvain Lebresne wr= ote: > >> >>> >>>> I still have some questions regarding the mapping. Please bear with me >>>> if these are stupid questions. I am quite new to Cassandra. >>>> >>>> The basic cassandra data model for a keyspace is something like this, >>>> right? >>>> >>>> SortedMap> >>>> ^ row key. determines which server(s) the rest is >>>> stored on >>>> ^ column key >>>> ^ >>>> timestamp (latest one wins) >>>> >>>> ^ value (can be size 0) >>>> >>> >>> It's a reasonable way to think of how things are stored internally, yes= . >>> Though as DuyHai mentioned, the first map is really sorting by token an= d in >>> general that means you use mostly the sorting of the second map concret= ely. >>> >>> >> Yes, understood. >> >> So the first SortedMap is sorted on some kind of hash of the actual key >> to make sure the data gets evenly distributed along the nodes? What if m= y >> key is already a good hash: is there a way to use an identity function a= s a >> hash function (in CQL)? >> > > It's possible, yes. The hash function we're talking about is what > Cassandra calls "the partitioner". You configure the partitioner in the > yaml config file and there is one partitioner, ByteOrderedPartitioner, th= at > is basically the identify function. > We however usually discourage user for using it because the partitioner i= s > global to a cluster and cannot be changed (you basically pick it at clust= er > creation time and are stuck with it until the end of time), and since > ByteOrderedPartitioner can easily lead to hotspot in the data distributio= n > if you're not careful...For those reasons, the default partitioner is als= o > much more tested, and I can't remember anyone mentioning the partitioner > has been a bottleneck. > > Thanks for the info. I thought that this might be possible to adjust on a per-keyspace level. But if you can only do this globally, then I will leave it alone. Other than the (probably negibile) performance impact of hashing the hash again, there is nothing wrong with doing so. Hashing a SHA1-hash will give a good distribution. anyway, this is getting a bit off-topic. cheers, R=FCdiger --047d7b66f0bd220cef04f329c923 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Mon, Feb 24, 2014 at 11:47 AM, Sylvain Lebresne <sy= lvain@datastax.com> wrote:
=
=A0

I still have some questions regarding the ma= pping. Please bear with me if these are stupid questions. I am quite new to Cassandra.

The = basic cassandra data model for a keyspace is something like this, right?
SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>><= br>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ^ row key. d= etermines which server(s) the rest is stored on
=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ^ column key
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ^ timestamp (latest one wi= ns)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 ^ value (can be size 0)

It's a reasonable wa= y to think of how things are stored internally, yes. Though as DuyHai menti= oned, the first map is really sorting by token and in general that means yo= u use mostly the sorting of the second map concretely.
=A0
Yes, understo= od.

So the first SortedMap is sorted on some kind of hash of the ac= tual key to make sure the data gets evenly distributed along the nodes? Wha= t if my key is already a good hash: is there a way to use an identity funct= ion as a hash function (in CQL)?

It's pos= sible, yes. The hash function we're talking about is what Cassandra cal= ls "the partitioner". You configure the partitioner in the yaml c= onfig file and there is one partitioner,=A0ByteOrderedPartitioner, that is = basically the identify function.
We however usually discourage user for using it because the partitione= r is global to a cluster and cannot be changed (you basically pick it at cl= uster creation time and are stuck with it until the end of time), and since= ByteOrderedPartitioner can easily lead to hotspot in the data distribution= if you're not careful...For those reasons, the default partitioner is = also much more tested, and I can't remember anyone mentioning the parti= tioner has been a bottleneck.

Thanks for the info. I t= hought that this might be possible to adjust on a per-keyspace level.
<= br>But if you can only do this globally, then I will leave it alone. Other = than the (probably negibile) performance impact of hashing the hash again, = there is nothing wrong with doing so. Hashing a SHA1-hash will give a good = distribution.

anyway, this is getting a = bit off-topic.

cheers,

R=FCd= iger
--047d7b66f0bd220cef04f329c923--