From user-return-33739-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Apr 26 01:51:02 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0CBB7F0CD for ; Fri, 26 Apr 2013 01:51:02 +0000 (UTC) Received: (qmail 89257 invoked by uid 500); 26 Apr 2013 01:50:59 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 89227 invoked by uid 500); 26 Apr 2013 01:50:59 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 89216 invoked by uid 99); 26 Apr 2013 01:50:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Apr 2013 01:50:59 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a52.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Apr 2013 01:50:54 +0000 Received: from homiemail-a52.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a52.g.dreamhost.com (Postfix) with ESMTP id 7E6F56B82F3 for ; Thu, 25 Apr 2013 18:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=tspYarrx/grvdIpfO+/9teWuWz o=; b=o7W38r2QfhOd+sPx0C5hqqAqQgHlpnLYlrz4H1Ovp75/3u+QW8CzK67lM8 845tfD7C8rB6OQOmg6e+2IOqxfZCmeR/FwzpsjHz3Lj9yopIUorAVJ4dY/Vi+y2f 2xmhnjz/np/NqC+KS5uIIwPtc70Su/MVOnenowtB59b48FBfk= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a52.g.dreamhost.com (Postfix) with ESMTPSA id 056E56B8269 for ; Thu, 25 Apr 2013 18:50:33 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_7A7823B9-E58E-40AC-B521-D42816FD4BAD" Message-Id: <39D73F09-23D5-45A5-8DEF-B45BBDFC4BE8@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Really odd issue (AWS related?) Date: Fri, 26 Apr 2013 13:50:32 +1200 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1503) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_7A7823B9-E58E-40AC-B521-D42816FD4BAD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > The messages appear right after the node "wakes up". Are you tracking CPU steal ?=20 ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 25/04/2013, at 4:15 AM, Robert Coli wrote: > On Wed, Apr 24, 2013 at 5:03 AM, Michael Theroux = wrote: >> Another related question. Once we see messages being dropped on one = node, our cassandra client appears to see this, reporting errors. We = use LOCAL_QUORUM with a RF of 3 on all queries. Any idea why clients = would see an error? If only one node reports an error, shouldn't the = consistency level prevent the client from seeing an issue? >=20 > If the client is talking to a broken/degraded coordinator node, RF/CL > are unable to protect it from RPCTimeout. If it is unable to > coordinate the request in a timely fashion, your clients will get > errors. >=20 > =3DRob --Apple-Mail=_7A7823B9-E58E-40AC-B521-D42816FD4BAD Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 The messages appear right after the node = "wakes up".
Are you tracking CPU steal = ? 

http://www.thelastpickle.com

On 25/04/2013, at 4:15 AM, Robert Coli <rcoli@eventbrite.com> = wrote:

On Wed, Apr 24, 2013 at 5:03 AM, Michael Theroux <mtheroux2@yahoo.com> = wrote:
Another related question.  Once = we see messages being dropped on one node, our cassandra client appears = to see this, reporting errors.  We use LOCAL_QUORUM with a RF of 3 = on all queries.  Any idea why clients would see an error?  If = only one node reports an error, shouldn't the consistency level prevent = the client from seeing an issue?

If the client is = talking to a broken/degraded coordinator node, RF/CL
are unable to = protect it from RPCTimeout. If it is unable to
coordinate the request = in a timely fashion, your clients will = get
errors.

=3DRob

= --Apple-Mail=_7A7823B9-E58E-40AC-B521-D42816FD4BAD--