Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 18F38D4AD for ; Tue, 9 Oct 2012 23:01:01 +0000 (UTC) Received: (qmail 42217 invoked by uid 500); 9 Oct 2012 23:00:58 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 42195 invoked by uid 500); 9 Oct 2012 23:00:58 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 42184 invoked by uid 99); 9 Oct 2012 23:00:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Oct 2012 23:00:58 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a58.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Oct 2012 23:00:50 +0000 Received: from homiemail-a58.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTP id ABF8F7D806A for ; Tue, 9 Oct 2012 16:00:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=W31GHfqaJA2IecXc+wwBXFYbra 4=; b=S2Ppfjc6Evslq+QrjG+hJd03GsT4OaQpovzi3ynM67ssS+i2Wazm+wltgm aQ0x4vHyD2hJNtLhmQfuzTFdahqnTgWlXkdtKi6BjwM98JS6q0tUK5bSQGejKxnX i/JMASeIj49of/vQT8lciW5YjTjiBSI+sh7HxIMmdvTXrg96U= Received: from [192.168.0.159] (unknown [116.90.132.105]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTPSA id 7FE427D805B for ; Tue, 9 Oct 2012 16:00:27 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_583ADA12-A7BA-47AB-8C00-1988E0CFFD96" Message-Id: <19C8179E-1594-4190-AB19-4F57328B03AF@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Subject: Re: READ messages dropped Date: Wed, 10 Oct 2012 12:00:24 +1300 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1498) --Apple-Mail=_583ADA12-A7BA-47AB-8C00-1988E0CFFD96 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > or how to solve it? Simple solution is move to m1.xlarge :) > In the last 3 days I see many messages of "READ messages dropped in = last 5000ms" on one of my 3 nodes cluster. The node is not able to keep up with the load.=20 Possible causes include excessive GC, aggressive compaction, or simply = too many requests. it also a good idea to take a look at iostat to see if the disk is = keeping up.=20 Hope that helps=20 =20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 9/10/2012, at 9:08 AM, Tamar Fraenkel wrote: > Hi! > In the last 3 days I see many messages of "READ messages dropped in = last 5000ms" on one of my 3 nodes cluster. > I see no errors in the log. > There are also messages of "Finished hinted handoff of 0 rows to = endpoint" but I had those for a while now, so I don't know if they are = related. > I am running Cassandra 1.0.8 on a 3 node cluster on EC2 m1.large = instances. Rep factor 3 (Quorum read and write) >=20 > Does anyone have a clue what I should be looking for, or how to solve = it? > Thanks, >=20 > Tamar Fraenkel=20 > Senior Software Engineer, TOK Media=20 >=20 > >=20 > tamar@tok-media.com > Tel: +972 2 6409736=20 > Mob: +972 54 8356490=20 > Fax: +972 2 5612956=20 >=20 >=20 >=20 --Apple-Mail=_583ADA12-A7BA-47AB-8C00-1988E0CFFD96 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
or how to solve = it?
Simple solution is move to m1.xlarge = :)

In the = last 3 days I see many messages of "READ messages dropped in last = 5000ms" on one of my 3 nodes cluster.
The node is = not able to keep up with the = load. 

Possible causes include excessive = GC, aggressive compaction, or simply too many = requests.

it also a good idea to take a look at = iostat to see if the disk is keeping = up. 

Hope that = helps 
 
=
http://www.thelastpickle.com

On 9/10/2012, at 9:08 AM, Tamar Fraenkel <tamar@tok-media.com> = wrote:

Hi!
In the last 3 days I see many = messages of "READ messages dropped in last 5000ms" on one of my 3 nodes = cluster.
I see no errors in the log.
There are also messages of = "Finished hinted handoff of 0 rows to endpoint" but I had those for a = while now, so I don't know if they are related.
I am running Cassandra 1.0.8 on a 3 node cluster on EC2 m1.large = instances. Rep factor 3 (Quorum read and write)

Does anyone have = a clue what I should be looking for, or how to solve = it?
Thanks,

Tamar Fraenkel 
Senior = Software Engineer, TOK Media 

<tokLogo.png>




= --Apple-Mail=_583ADA12-A7BA-47AB-8C00-1988E0CFFD96--