Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 64BA6F5D9 for ; Thu, 28 Mar 2013 09:30:08 +0000 (UTC) Received: (qmail 68797 invoked by uid 500); 28 Mar 2013 09:14:13 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 68599 invoked by uid 500); 28 Mar 2013 09:14:04 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 68535 invoked by uid 99); 28 Mar 2013 09:14:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Mar 2013 09:14:02 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of arodrime@gmail.com designates 209.85.215.52 as permitted sender) Received: from [209.85.215.52] (HELO mail-la0-f52.google.com) (209.85.215.52) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Mar 2013 09:13:58 +0000 Received: by mail-la0-f52.google.com with SMTP id fs12so17359583lab.11 for ; Thu, 28 Mar 2013 02:13:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=+DGoeajzh5ftz1Zocxy3KZ0TjJE2TASlZsVoA8a1NqU=; b=xuo0hw5PaBRWxYKrpxiptuShL8Muvfn5+M39ghEkm6TCME6abjtBKxEe1EJzzOaraq JLp2xjzMo5AGahP/5oyOrfEpPY+Ea7jqRaWDxVXWOG0SnylHCYLFSYECdnf+CouDNX+Q MyS77R8IQGtVjpbOUow6tXExUF6/EvfGGSF43PmHYglQfbCIbEIpYW3UA5iSKTXpUB0w c/FjwnIxIBInrl/Q/915ZpxIOIzalPKeeuJf5Y7rvm18etH2Tk+zYlhsNuaJu9sus7hb 0mFRmllZD6NeshYlF6z7QlPdLliaNASOoSc09n7G9DkPd6JwaeEpMmm2RKcuqD/Bdjxj 1Lcw== X-Received: by 10.112.26.202 with SMTP id n10mr12003063lbg.15.1364462016698; Thu, 28 Mar 2013 02:13:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.44.230 with HTTP; Thu, 28 Mar 2013 02:13:16 -0700 (PDT) In-Reply-To: References: From: Alain RODRIGUEZ Date: Thu, 28 Mar 2013 10:13:16 +0100 Message-ID: Subject: Re: weird behavior with RAID 0 on EC2 To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec55555fed0a22804d8f8910a X-Virus-Checked: Checked by ClamAV on apache.org --bcaec55555fed0a22804d8f8910a Content-Type: text/plain; charset=ISO-8859-1 Ok, if you're going to look into it, please keep me/us posted. It happen twice for me, the same day, within a few hours on the same node and only happened to 1 node out of 12, making this node almost unreachable. 2013/3/28 aaron morton > I noticed this on an m1.xlarge (cassandra 1.1.10) instance today as well, > 1 or 2 disks in a raid 0 running at 85 to 100% the others 35 to 50ish. > > Have not looked into it. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 26/03/2013, at 11:57 PM, Alain RODRIGUEZ wrote: > > We use C* on m1.xLarge AWS EC2 servers, with 4 disks xvdb, xvdc, xvdd, > xvde parts of a logical Raid0 (md0). > > I use to see their use increasing in the same way. This morning there was > a normal minor compaction followed by messages dropped on one node (out of > 12). > > Looking closely at this node I saw the following: > > http://img69.imageshack.us/img69/9425/opscenterweirddisk.png > > On this node, one of the four disks (xvdd) started working hardly while > other worked less intensively. > > This is quite weird since I always saw this 4 disks being used the exact > same way at every moment (as you can see on 5 other nodes or when the node > ".239" come back to normal). > > Any idea on what happened and on how it can be avoided ? > > Alain > > > --bcaec55555fed0a22804d8f8910a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Ok, if you're going to look into it, please keep me/us= posted.

It happen twice for me, the same day, wit= hin a few hours on the same node and only happened to 1 node out of 12, mak= ing this node almost unreachable.


2013/3/= 28 aaron morton <aaron@thelastpickle.com>
I noticed this on an m1.xlarge (cassand= ra 1.1.10) instance today as well, 1 or 2 disks in a raid 0 running at 85 t= o 100% the others 35 to 50ish.=A0

Have not looked into i= t.=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 26/03/2013, at 11:57 PM, Alain RODRIGUEZ <arodrime@gmail.com> wrote= :

We use C* on m1.xLarg= e AWS EC2 servers, with 4 disks xvdb, xvdc, xvdd, xvde parts of a logical R= aid0 (md0).

I use to see their use increasing in the same way. This= morning there was a normal minor compaction followed by messages dropped o= n one node (out of 12).

Looking closely at this node I saw the following:
=


On this node, one of the four disks (xvdd) starte= d working hardly while other worked less intensively.

<= div>This is quite weird since I always saw this 4 disks being used the exac= t same way at every moment (as you can see on 5 other nodes or when=A0the n= ode ".239"=A0come back to normal).

Any idea on what happened and on how it can be avoided = ?

Alain


--bcaec55555fed0a22804d8f8910a--