Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71BAAF78B for ; Thu, 28 Mar 2013 02:27:14 +0000 (UTC) Received: (qmail 29114 invoked by uid 500); 28 Mar 2013 02:27:11 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 29006 invoked by uid 500); 28 Mar 2013 02:27:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 28988 invoked by uid 99); 28 Mar 2013 02:27:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Mar 2013 02:27:11 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a94.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Mar 2013 02:27:05 +0000 Received: from homiemail-a94.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTP id 610F238A071 for ; Wed, 27 Mar 2013 19:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=gVjCcmG0vuJgzyErk7zbjTK03d w=; b=Aa/o1etzKE5E3rTNJIWfjuAnFdf1oYnYVi9+cT3hXAQ5o5od8jJz38Bm8x htfgkxS+LWDzX99zHlHzdyTuq3nYraHnHfR9rghEthd5uBbmUPbC+qjqmPQvOC7L mZnEy6gRFwPMrFhPvjbipHqNW7VSy2BD+rR98teoNmMF5eE5o= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTPSA id D2A7638A06F for ; Wed, 27 Mar 2013 19:26:41 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_A05BE8CD-3D8F-41B0-A26E-835BA387149B" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: weird behavior with RAID 0 on EC2 Date: Thu, 28 Mar 2013 15:26:40 +1300 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_A05BE8CD-3D8F-41B0-A26E-835BA387149B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 I noticed this on an m1.xlarge (cassandra 1.1.10) instance today as = well, 1 or 2 disks in a raid 0 running at 85 to 100% the others 35 to = 50ish.=20 Have not looked into it.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 26/03/2013, at 11:57 PM, Alain RODRIGUEZ wrote: > We use C* on m1.xLarge AWS EC2 servers, with 4 disks xvdb, xvdc, xvdd, = xvde parts of a logical Raid0 (md0). >=20 > I use to see their use increasing in the same way. This morning there = was a normal minor compaction followed by messages dropped on one node = (out of 12). >=20 > Looking closely at this node I saw the following: >=20 > http://img69.imageshack.us/img69/9425/opscenterweirddisk.png >=20 > On this node, one of the four disks (xvdd) started working hardly = while other worked less intensively. >=20 > This is quite weird since I always saw this 4 disks being used the = exact same way at every moment (as you can see on 5 other nodes or when = the node ".239" come back to normal). >=20 > Any idea on what happened and on how it can be avoided ? >=20 > Alain --Apple-Mail=_A05BE8CD-3D8F-41B0-A26E-835BA387149B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 I = noticed this on an m1.xlarge (cassandra 1.1.10) instance today as well, = 1 or 2 disks in a raid 0 running at 85 to 100% the others 35 to = 50ish. 

Have not looked into = it. 

Cheers

http://www.thelastpickle.com

On 26/03/2013, at 11:57 PM, Alain RODRIGUEZ <arodrime@gmail.com> = wrote:

We use C* on m1.xLarge AWS EC2 servers, = with 4 disks xvdb, xvdc, xvdd, xvde parts of a logical Raid0 = (md0).

I use to see their use = increasing in the same way. This morning there was a normal minor = compaction followed by messages dropped on one node (out of 12).

Looking closely at this node I = saw the following:


On this node, one of the = four disks (xvdd) started working hardly while other worked less = intensively.

This is = quite weird since I always saw this 4 disks being used the exact same = way at every moment (as you can see on 5 other nodes or when the = node ".239" come back to normal).

Any idea on what happened and = on how it can be avoided ?

Alain

= --Apple-Mail=_A05BE8CD-3D8F-41B0-A26E-835BA387149B--