Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 971A89191 for ; Wed, 23 May 2012 01:11:58 +0000 (UTC) Received: (qmail 84967 invoked by uid 500); 23 May 2012 01:11:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84937 invoked by uid 500); 23 May 2012 01:11:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84927 invoked by uid 99); 23 May 2012 01:11:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 May 2012 01:11:56 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of koji.lin@gmail.com designates 209.85.215.44 as permitted sender) Received: from [209.85.215.44] (HELO mail-lpp01m010-f44.google.com) (209.85.215.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 May 2012 01:11:52 +0000 Received: by lagv3 with SMTP id v3so5457850lag.31 for ; Tue, 22 May 2012 18:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=J4lomDCHw4Vv6duZd60qcdw8KJi9EKwTMIgOyhdaMRA=; b=uKmr6mKmOfa0KgRichVqHLVhsaanWr22a32Nn7XjtMXhCfJ9xfVvhhnxqxt1yKCqKi oo37se3FoaixyzwQ34ehX6gOm932TyTAXNFe0nRlhhXEa3f+5aGt9hYieGH/VeA3/L3l Bge4ekGAaGb6n5Rcv+WWmkCLsHj6PYmca1WLWwvQ6lSvAUjUOmwGIvEi3AbGlqp8WZRe BWio66gjiWDC5o4D2TDVgcLSFIhI8E4PHEgfCLOeJLo6Lh0FTbUyN46quAGyJ20iuqxq n+efBIoNBnYQ4/pcQVLmygEsbMRgd2OT8lUX28A88I7LPrJ6J2Br7E5Kz6O752v8SLZ1 W9fA== Received: by 10.152.111.33 with SMTP id if1mr24991981lab.10.1337735490325; Tue, 22 May 2012 18:11:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.103.72 with HTTP; Tue, 22 May 2012 18:11:10 -0700 (PDT) In-Reply-To: <4FBC1A84.8070202@syncopated.net> References: <4FBC1A84.8070202@syncopated.net> From: koji Lin Date: Wed, 23 May 2012 09:11:10 +0800 Message-ID: Subject: Re: Using EC2 ephemeral 4disk raid0 cause high iowait trouble To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d04083ad7b44bb804c0a9d009 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04083ad7b44bb804c0a9d009 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Thanks for your information, we will try that. koji 2012/5/23 Deno Vichas > for what it's worth i've been having pretty good success using the > Datastax AMIs. > > > > On 5/17/2012 6:59 PM, koji Lin wrote: > > Hi > > We use amazon ami 3.2.12-3.2.4.amzn1.x86_64 > > and some of our data file are more than 10G > > thanks > > koji > 2012-5-16 =E4=B8=8B=E5=8D=886:00 =E6=96=BC "aaron morton" =E5=AF=AB=E9=81=93=EF=BC=9A > >> On Ubuntu ? Sounds like http://wiki.apache.org/cassandra/FAQ#ubuntu_hang= s >> >> Cheers >> >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 16/05/2012, at 2:13 PM, koji Lin wrote: >> >> Hi >> >> Our service already run cassandra 1.0 on 1x ec2 instances(with ebs), and >> we saw lots of discussion talk about using ephemeral raid for better >> performance and consistent performance. >> >> So we want to create new instance using 4 ephemeral raid0, and copy the >> data from ebs to finally replace the old instance and reduce some . >> >> we create the xlarge instance with -b '/dev/sdb=3Dephemeral0' -b >> '/dev/sdc=3Dephemeral1' -b '/dev/sdd=3Dephemeral2' -b '/dev/sde=3Depheme= ral3', >> >> and use mdadm command like this mdadm --create /dev/md0 --level=3D0 -c2= 56 >> --raid-devices=3D4 /dev/sdb /dev/sdc /dev/sdd /dev/sde >> >> after copying file and start the cassandra(same token as old instance it >> replaced). >> >> we saw the read is really fast always keep 2xxm/sec, but system load >> exceed 40, with high iowait, and lots of client get timeout result. We >> guess maybe it's the problem of ec2 instance, so we create another one w= ith >> same setting to replace other machine ,but the result is same . Then we >> rollback to ebs with single disk ,read speed keeps at 1xmb/sec but syste= m >> becomes well .(using ebs with 2 disks raid0 will keep at 2xmb/sec and >> higher iowait then single disk ,but still works) >> >> Is there anyone meet the same problem too ? or do we forget something to >> configure? >> >> thank you >> >> koji >> >> >> > --=20 Koji Lin blog: http://www.javaworld.com.tw/roller/koji/ --f46d04083ad7b44bb804c0a9d009 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi
Thanks for your information, we will try that.

<= div>koji

2012/5/23 Deno Vichas <deno@= syncopated.net>
=20 =20 =20
for what it's worth i've been having pretty good success using = the Datastax AMIs.



On 5/17/2012 6:59 PM, koji Lin wrote:

Hi

We use amazon ami 3.2.12-3.2.4.amzn1.x86_64

and some of our data file are more than 10G

thanks

koji

2012-5-16 =E4=B8=8B=E5=8D=886:00 =E6=96=BC= "aaron morton" <aaron@thelastpickle.com> =E5=AF=AB=E9=81=93=EF=BC=9A
On Ubuntu ? Sounds like=C2=A0= http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 16/05/2012, at 2:13 PM, koji Lin wrote:

Hi

Our service already run cassandra 1.0 on 1x ec2 instances(with ebs), and we saw lots of discussion talk about using=C2=A0 ephemeral=C2=A0raid for better performance and consistent performance.

So we want to create new instance using 4=C2=A0ephemer= al raid0, and copy the data from ebs to finally replace the old instance and reduce some .=C2=A0

we create the xlarge instance with -b '/dev/sdb=3Dephemeral0' -b '/dev/sdc=3Dephe= meral1' -b '/dev/sdd=3Dephemeral2' -b '/dev/sde=3Dephe= meral3',

and use mdadm command like this=C2=A0=C2=A0mdadm --cre= ate /dev/md0 --level=3D0 -c256 --raid-devices=3D4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

after copying file and start the cassandra(same token as old instance it replaced).

we saw the read is really fast always keep 2xxm/sec, but system load exceed 40, with high iowait, and lots of client get timeout result. We guess maybe it's the problem of ec2 instance, so we create another one with same setting to replace other machine ,but the result is same . Then we rollback to ebs with single disk ,read speed keeps at 1xmb/sec but system becomes well .(using ebs with 2 disks raid0 will keep at 2xmb/sec and higher iowait then single disk ,but still works)

Is there anyone meet the same problem too ? or do we forget something to configure?

thank you

koji






--
Koji Lin
blog:=C2=A0http://www.javaworld.com= .tw/roller/koji/
--f46d04083ad7b44bb804c0a9d009--