Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0A2329420 for ; Thu, 5 Jan 2012 19:21:07 +0000 (UTC) Received: (qmail 69585 invoked by uid 500); 5 Jan 2012 19:21:04 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 69538 invoked by uid 500); 5 Jan 2012 19:21:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 69523 invoked by uid 99); 5 Jan 2012 19:21:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jan 2012 19:21:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a46.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jan 2012 19:20:55 +0000 Received: from homiemail-a46.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a46.g.dreamhost.com (Postfix) with ESMTP id 887CD3E4073 for ; Thu, 5 Jan 2012 11:20:22 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=IYgX1dAo/f igy0UPo6+2Z0MZFxNVU4Gs1+KILr/+aJVwOFHARUJPgwlK7RK39H05VkP1tmkrwg 2IdjCDygKyHmWy/a5748d5La1hRsEkqc+nevLPgGhws7BVQMnfYEgFJilO2lLE9y l7KufacqO2huw3roxm9FWqrDMm/x0Xil4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=RUzf2W6kRV0XjcR4 lHGiFMFGZe4=; b=CA0dkAmPPO/nbbR0a0pqJkC6frbkj6pVAgqZ/H1ztr2K24KN HdocpSP1alzojP0NVGzQKzMK3nKygTXDlvXglfhlc5TIz1DpDduBtCIaV2bcDWyX tP5rgiJB6pqXTkxHbLIJ2w5HaX6nek1Pw3oP1RHwkyOMvqRIiXkoa5U1CHE= Received: from [172.16.1.4] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a46.g.dreamhost.com (Postfix) with ESMTPSA id 089683E406A for ; Thu, 5 Jan 2012 11:20:09 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: multipart/alternative; boundary="Apple-Mail=_06EB121E-E524-4950-A54C-083BB113CA20" Subject: Re: Writes slower then reads Date: Fri, 6 Jan 2012 08:20:06 +1300 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <020FFBD7-FAEF-4053-8BDD-424E7DB574A2@thelastpickle.com> X-Mailer: Apple Mail (2.1251.1) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_06EB121E-E524-4950-A54C-083BB113CA20 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 What happens when you turn off the cron jobs ?=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 6/01/2012, at 6:57 AM, Philippe wrote: > Unless you are doing huge batches no... don't have any other idea for = now... >=20 > 2012/1/5 R. Verlangen > The write and read load is very minimal the moment. Roughly 10 writes = + 10 reads / second. So 20 operations per second. Don't think that = overloads my cluster, does it? >=20 >=20 > 2012/1/5 Philippe > You may be overloading the cluster though... >=20 > My hypothesis is that your traffic is being spread across your node = and that one slow node is slowing down the fraction of traffic that goes = to that node (when it's acting as coordinator). > So what I would do is reduce the read load a lot to make sure I don't = overload the cluster and measure if I see a 1/RF improvement in response = time which would validate my hypothesis. >=20 >=20 > 2012/1/5 R. Verlangen >=20 > It does not appear to affect the response time, certainly not in a = positive way. >=20 >=20 > 2012/1/5 Philippe > What if you shutdown the cassandra service on the slow node, does that = improve your read performance ? > If it does then that sole node is responsible for the slow down = because it can't act as a coordinator fast enough. >=20 > 2012/1/5 R. Verlangen >=20 > I'm also reading with CL =3D ONE >=20 >=20 > 2012/1/5 Philippe > Depending on the CL you're reading at it will yes : if the CL requires = that the "slow" node create a digest of the data and send it to the = coordinator then it might explain the poor performance on reads. What is = your read CL ? >=20 > 2012/1/5 R. Verlangen >=20 > As I posted this I noticed that the other node's CPU is running high = on some other cronjobs (every couple of minutes to 60% usage). Is the = lack of more CPU cycles a problem in this case? >=20 > Robin >=20 > 2012/1/5 R. Verlangen >=20 > CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. = Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / = out). Paging 0. System int ~ 1300, csw ~ 2500. >=20 >=20 > 2012/1/5 Philippe > What can you see in vmstat/dstat ? >=20 > Le 5 janv. 2012 11:58, "R. Verlangen" a =E9crit : >=20 > Hi there, >=20 > I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF =3D = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk = throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with = 16GB ram).=20 >=20 > When I'm running my writes against the cluster with cl =3D ONE all = reads appear to be faster then the writes.=20 >=20 > Average write speed =3D 1600us/operation > Average read speed =3D 200us/operation >=20 > I'm really wondering why this is the case. Anyone got a clue? >=20 > With kind regards, > Robin=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 --Apple-Mail=_06EB121E-E524-4950-A54C-083BB113CA20 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 What = happens when you turn off the cron jobs = ? 

Cheers

http://www.thelastpickle.com

On 6/01/2012, at 6:57 AM, Philippe wrote:

Unless you = are doing huge batches no... don't have any other idea for = now...

2012/1/5 R. Verlangen <robin@us2.nl>
The write and read load is very minimal the moment. Roughly 10 writes + = 10 reads / second. So 20 operations per second. Don't think that = overloads my cluster, does it?


2012/1/5 Philippe <watcherfr@gmail.com>
You may be overloading = the cluster though...

My hypothesis is that your traffic is = being spread across your node and that one slow node is slowing down the = fraction of traffic that goes to that node (when it's acting as = coordinator).
So what I would do is reduce the read load a lot to make sure I = don't overload the cluster and measure if I see a 1/RF improvement in = response time which would validate my = hypothesis.


2012/1/5 R. Verlangen <robin@us2.nl>

It does not appear to affect the response time, certainly not in a = positive way.


2012/1/5 = Philippe <watcherfr@gmail.com>
What if you shutdown = the cassandra service on the slow node, does that improve your read = performance ?
If it does then that sole node is responsible for the = slow down because it can't act as a coordinator fast enough.

2012/1/5 R. Verlangen <robin@us2.nl>

I'm also reading with CL =3D ONE


2012/1/5 Philippe <watcherfr@gmail.com>
Depending on the CL you're reading at it will yes : if the CL requires = that the "slow" node create a digest of the data and send it to the = coordinator then it might explain the poor performance on reads. What is = your read CL ?

2012/1/5 R. Verlangen <robin@us2.nl>

As I posted this I noticed that the other node's CPU is running high on = some other cronjobs (every couple of minutes to 60% usage). Is the lack = of more CPU cycles a problem in this = case?

Robin

2012/1/5 R. Verlangen <robin@us2.nl>

CPU is idle (< 10% usage). Disk reads occasionally blocks over = 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s = (in / out). Paging 0. System int ~ 1300, csw ~ 2500.


2012/1/5 Philippe <watcherfr@gmail.com>

What can you see in vmstat/dstat ?

Le 5 janv. 2012 11:58, "R. Verlangen" <robin@us2.nl> a = =E9crit :

Hi there,

I'm running a cassandra 0.8.6 cluster with = 2 nodes (in 2 DC's), RF =3D 2. Actual data on the nodes is only 1GB. = Disk latency < 1ms. Disk throughput ~ 0.4MB/s. OS load always below 1 = (on a 8 core machine with 16GB ram). 

When I'm running my writes against the cluster with = cl =3D ONE all reads appear to be faster then the = writes. 

Average write speed =3D = 1600us/operation
Average read speed =3D 200us/operation

I'm really wondering why this is the case. Anyone = got a clue?

With kind = regards,
Robin 










= --Apple-Mail=_06EB121E-E524-4950-A54C-083BB113CA20--