Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D58767D45 for ; Tue, 15 Nov 2011 21:23:08 +0000 (UTC) Received: (qmail 24656 invoked by uid 500); 15 Nov 2011 21:23:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 24627 invoked by uid 500); 15 Nov 2011 21:23:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 24619 invoked by uid 99); 15 Nov 2011 21:23:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Nov 2011 21:23:06 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of watcherfr@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-wy0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Nov 2011 21:22:58 +0000 Received: by wyf28 with SMTP id 28so6552580wyf.31 for ; Tue, 15 Nov 2011 13:22:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=thNK/RjRhW9uKW6GU847we4+PEFQhpKZL14Er8XZur8=; b=c/zuMp4mydbj8CwvZLOePYc6mGA6BatzuaVPfny4MzR9cSgJY7KbaKhHeGHFwGL4Y4 Z9zIcum7gwAQZ2H1NXR+gCr+BTqIX9jHClozMDvDuOO+Stv7VK5d1SVzkEBnfYPb+eSw tvLtgHSpyGXXICgFLRrZi9Gpp6U1xNV129Hgs= MIME-Version: 1.0 Received: by 10.180.80.98 with SMTP id q2mr32029294wix.53.1321392158343; Tue, 15 Nov 2011 13:22:38 -0800 (PST) Received: by 10.180.81.228 with HTTP; Tue, 15 Nov 2011 13:22:38 -0800 (PST) In-Reply-To: References: Date: Tue, 15 Nov 2011 22:22:38 +0100 Message-ID: Subject: Re: Network traffic patterns From: Philippe To: user Content-Type: multipart/alternative; boundary=f46d0442834635060f04b1cc96eb X-Virus-Checked: Checked by ClamAV on apache.org --f46d0442834635060f04b1cc96eb Content-Type: text/plain; charset=ISO-8859-1 Sorry about the previous message, I've enabled keyboard shortcuts on gmail...*sigh*... Hello, I'm trying to understand the network usage I am seeing in my cluster, can anyone shed some light? It's an RF=3, 12-node, cassandra 0.8.6 cluster. repair is performed on each node once a week, with a rolling schedule. The nodes are p13,p14,p15...p24 and are consecutive in that order on the ring. Each node is only a cassandra database. I am hitting the cluster from another server (p4). p4 is doing this with 20 threads in parallel 1. read a lot of data (some columns for hundreds to tens of thousands of keys, split into 512-key multigets) 2. process the data 3. write back a byte array to cassandra (average size is 400 bytes) 4. go back to 1 According to my munin graphs, network usage is about as follows. I am not surprised at the bias towards p13-p15 as p4 is getting & storing data mainly for keys located on one of those nodes. - p4 : 1.5Mb/s in and out - p13-p15 : 15Mb/s in and 80Mb/s out - p16-p24 : 45Mb/s in and 5Mb/s out What I don't understand is why p4 is only seeing 1.5Mb/s while I see 80Mb/s on p13 & p15. The way I understand this: - p4 makes a multiget to the cluster, electing to use any node in the cluster (IN traffic for describe the query) - coordinator node replays the query on all 3 replicas (so 3 servers each get the IN traffic, mostly p13-p15) - each server replies to coordinator - coordinator chooses matching values and sends back data to p4 So if p13-p15 are outputting 80Mb/s why am I not seeing 80Mb/s coming into p4 which is on the receiving end ? Thanks 2011/11/15 Philippe > Hello, > I'm trying to understand the network usage I am seeing in my cluster, can > anyone shed some light? > It's an RF=3, 12-node, cassandra 0.8.6 cluster. The nodes are > p13,p14,p15...p24 and are consecutive in that order on the ring. > Each node is only a cassandra database. I am hitting the cluster from > another server (p4). > > The pattern on p4 is the pattern is to > > 1. read a lot of data (some columns for hundreds to tens of thousands > of keys, split into 512-key multigets) > 2. process the data > 3. write back a byte array to cassandra (average size is 400 bytes) > > > p4 reads as > --f46d0442834635060f04b1cc96eb Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Sorry about the previous message, I've enabled keyboard shortcuts on gm= ail...*sigh*...

Hello,
I'm trying to understand the network usage I am seeing in my cluster, c= an anyone shed some light?
It's an RF= =3D3, 12-node, cassandra 0.8.6 cluster. repair is performed on each node on= ce a week, with a rolling schedule.
The nodes are p13,p14,p15...p24 and are consecu= tive in that order on the ring.=A0Each node is only a cassandra database. I= am hitting the cluster from another server (p4).

p4 is doi= ng this with 20 threads in parallel=A0
  1. read a lo= t of data (some columns for hundreds to tens of thousands of keys, split in= to 512-key multigets)
  2. process the data
  3. write back a byte array to cassandra (average size is 400 byt= es)
  4. go back to 1
According to my munin graphs, network usage is about as follows.=A0I am not= surprised at the bias towards p13-p15 as p4 is getting & storing data = mainly for keys located on one of those nodes.
  • p4 : 1.= 5Mb/s in and out
  • p13-p15 :=A015Mb/s in and=A080Mb/s out
  • p16-p24 : 45Mb/s in= and 5Mb/s out
What I don't understand is why p= 4 is only seeing 1.5Mb/s while I see 80Mb/s on p13 & p15.

The way I understand this:
  • p4 makes a multiget to the cluste= r, electing to use any node in the cluster (IN traffic for describe the que= ry)
  • coordinator= node replays the query on all 3 replicas (so 3 servers each get the IN tra= ffic, mostly p13-p15)
  • each server replies to coordinator
  • coordinator= chooses matching values and sends back data to p4
So if p13-p15 are = outputting 80Mb/s why am I not seeing 80Mb/s coming into p4 which is on the= receiving end ?

Tha= nks

2011/11/15 Philippe = <watcherfr@gmail.com>
Hello,
I'm trying to understand the network usage I am seeing in my= cluster, can anyone shed some light?
It's an RF=3D3, 12-node= , cassandra 0.8.6 cluster. The nodes are p13,p14,p15...p24 and are consecut= ive in that order on the ring.
Each node is only a cassandra database. I am hitting the cluster from = another server (p4).

The pattern on p4 is the patt= ern is to=A0
  1. read a lot of data (some columns for hundred= s to tens of thousands of keys, split into 512-key multigets)
  2. process the data
  3. write back a byte array to cassandra (average = size is 400 bytes)

p4 reads as

--f46d0442834635060f04b1cc96eb--