Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 71922 invoked from network); 2 Jun 2010 03:21:32 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 2 Jun 2010 03:21:32 -0000 Received: (qmail 65553 invoked by uid 500); 2 Jun 2010 03:21:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 65539 invoked by uid 500); 2 Jun 2010 03:21:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 65531 invoked by uid 99); 2 Jun 2010 03:21:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 03:21:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lwl.roger@gmail.com designates 209.85.212.44 as permitted sender) Received: from [209.85.212.44] (HELO mail-vw0-f44.google.com) (209.85.212.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 03:21:23 +0000 Received: by vws11 with SMTP id 11so2001414vws.31 for ; Tue, 01 Jun 2010 20:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=jwwsZ2E2ONJhKskZDC72RlB6Pk9/dThVuDgJUG/oYIU=; b=Tsl/tpThfn+jPbfrstk+uz9vG2xDjS9ADfHHTVKKZyTB0YbeqkL61h7vwiu+ntX9aL d7CbKrTy/JWFcRdBkS0PkmIi0I5urNd5VF1fwbufaKBA9envTt6/gbCHF4CJfPu4E3pY 9Jfw/h3EE8mndMNTCFsUwZY8NNGApM5hiS4ps= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=krjeSy54PEE/ELgtO9u7TA85XUGrsz5OI9Zsjx4vGj5DeALygWG5QyERL/FifqIqIE 3QJuRwlfec/dHyL6KzKHNzZnQvzF5zgk3vL111qq9AFW4OrQIJjjeAQfT7wd96gnOj1u gfjlxDQeYT6Q3z+YwDcgoQ2msSjBRUpVTyQGI= MIME-Version: 1.0 Received: by 10.220.108.34 with SMTP id d34mr5247394vcp.150.1275448859218; Tue, 01 Jun 2010 20:20:59 -0700 (PDT) Received: by 10.220.199.6 with HTTP; Tue, 1 Jun 2010 20:20:59 -0700 (PDT) In-Reply-To: <1275448496.2481.57.camel@supertool-2907-01> References: <1275382091.2481.36.camel@supertool-2907-01> <1275443316.2481.55.camel@supertool-2907-01> <1275444721.2481.56.camel@supertool-2907-01> <1275448496.2481.57.camel@supertool-2907-01> Date: Wed, 2 Jun 2010 11:20:59 +0800 Message-ID: Subject: Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing speed test From: lwl To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=00c09f8c24ad2ed3f504880394c7 X-Virus-Checked: Checked by ClamAV on apache.org --00c09f8c24ad2ed3f504880394c7 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable 3.32 concurrent read & 128 write in storage-conf.xml, other cache enlarged as well. -------------------- maybe you can try to decrease the size of caches. =D4=DA 2010=C4=EA6=D4=C22=C8=D5 =C9=CF=CE=E711:14=A3=ACShuai Yuan =D0=B4=B5=C0=A3=BA > =D4=DA 2010-06-02=C8=FD=B5=C4 10:37 +0800=A3=AClwl=D0=B4=B5=C0=A3=BA > > is all the 4 servers' MEM almost 100%? > > Yes > > > =D4=DA 2010=C4=EA6=D4=C22=C8=D5 =C9=CF=CE=E710:12=A3=ACShuai Yuan =D0=B4 > > =B5=C0=A3=BA > > Thanks lwl. > > > > Then is there anyway of tuning this, faster flush to disk or > > else? > > > > Cheers, > > > > Kevin > > > > =D4=DA 2010-06-02=C8=FD=B5=C4 09:57 +0800=A3=AClwl=D0=B4=B5=C0= =A3=BA > > > > > MEM: almost 100% (16GB) > > > ----------------- > > > maybe this is the bottleneck. > > > writing concerns Memtable and SSTable in memory. > > > > > > =D4=DA 2010=C4=EA6=D4=C22=C8=D5 =C9=CF=CE=E79:48=A3=ACShuai Y= uan > > =D0=B4 > > > =B5=C0=A3=BA > > > =D4=DA 2010-06-01=B6=FE=B5=C4 15:00 -0500=A3=ACJonath= an Shook=D0=B4=B5=C0=A3=BA > > > > Also, what are you meaning specifically by 'slow'? > > Which > > > measurements > > > > are you looking at. What are your baseline > > constraints for > > > your test > > > > system? > > > > > > > > > > Actually, the problem is the utilizaton of > > resources(for a > > > single > > > machine): > > > CPU: 700% / 1600% (16 cores) > > > MEM: almost 100% (16GB) > > > Swap: almost 0% > > > Disk IO(write): 20~30MB / 200MB (7.2k raid5, > > benchmarked > > > previously) > > > NET: up to 100Mbps / 950Mbps (1Gbps, tuned and > > benchmarked > > > previously) > > > > > > So the speed of generating load, about 15M/s as > > reported > > > before seems > > > quite slow to me. I assume the system should get at > > least > > > about 50MB/s > > > of Disk IO speed. > > > > > > MEM? I don't think it plays a major role in this > > writing game. > > > What's > > > the bottleneck of the system? > > > > > > P.S > > > about Consistency Level, I've tried ONE/DCQUORUM and > > found ONE > > > is about > > > 10-15% faster. However that's neither a promising > > result. > > > > > > Thanks! > > > > > > Kevin > > > > > > > > > > > 2010/6/1 =CA=B7=D3=A2=BD=DC : > > > > > Hi, It would be better if we know which > > Consistency Level > > > did you choose, > > > > > and what is the schema of test data? > > > > > > > > > > =D4=DA 2010=C4=EA6=D4=C21=C8=D5 =CF=C2=CE=E74:48= =A3=ACShuai Yuan > > > =D0=B4=B5=C0=A3=BA > > > > >> > > > > >> Hi all, > > > > >> > > > > >> I'm testing writing speed of cassandra with 4 > > servers. > > > I'm confused by > > > > >> the behavior of cassandra. > > > > >> > > > > >> ---env--- > > > > >> load-data app written in c++, using > > libcassandra (w/ > > > modified batch > > > > >> insert) > > > > >> 20 writing threads in 2 processes running on 2 > > servers > > > > >> > > > > >> ---optimization--- > > > > >> 1.turn log level to INFO > > > > >> 2.JVM has 8G heap > > > > >> 3.32 concurrent read & 128 write in > > storage-conf.xml, > > > other cache > > > > >> enlarged as well. > > > > >> > > > > >> ---result--- > > > > >> 1-monitoring by `date;nodetool -h host ring` > > > > >> I add all load together and measure the writing > > speed by > > > > >> (load_difference / time_difference), and I get > > about > > > 15MB/s for the > > > > >> whole cluster. > > > > >> > > > > >> 2-monitoring by `iostat -m 10` > > > > >> I can watch the disk_io from the system level > > and have > > > about 10MB/s - > > > > >> 65MB/s for a single machine. Very big variance > > over time. > > > > >> > > > > >> 3-monitoring by `iptraf -g` > > > > >> In this way I watch the communication between > > servers and > > > get about > > > > >> 10MB/s for a single machine. > > > > >> > > > > >> ---opinion--- > > > > >> So, have you checked the writing speed of > > cassandra? I > > > feel it's quite > > > > >> slow currently. > > > > >> > > > > >> Could anyone confirm this is the normal writing > > speed of > > > cassandra, or > > > > >> please provide someway of improving it? > > > > >> -- > > > > >> Kevin Yuan > > > > >> www.yuan-shuai.info > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Kevin Yuan > > > www.yuan-shuai.info > > > > > > > > > > > > > > > > > > > > > -- > > Shuai Yuan =D4=AC=CB=A7 > > Supertool Corp. =B1=B1=BE=A9=D1=A7=D6=AE=CD=BE=CD=F8=C2=E7=BF= =C6=BC=BC=D3=D0=CF=DE=B9=AB=CB=BE > > > > www.yuan-shuai.info > > > > > > > > -- > Shuai Yuan =D4=AC=CB=A7 > Supertool Corp. =B1=B1=BE=A9=D1=A7=D6=AE=CD=BE=CD=F8=C2=E7=BF=C6=BC=BC=D3= =D0=CF=DE=B9=AB=CB=BE > www.yuan-shuai.info > > > --00c09f8c24ad2ed3f504880394c7 Content-Type: text/html; charset=GB2312 Content-Transfer-Encoding: quoted-printable 3.32 concurrent read & 128= write in storage-conf.xml, other cache
enlarged as well.
--------------------

<= span class=3D"Apple-style-span" style=3D"border-collapse: collapse;">maybe = you can try to decrease the size of caches.

=D4=DA 2010=C4=EA6=D4=C22=C8= =D5 =C9=CF=CE=E711:14=A3=ACShuai Yuan <yuanshuai@supertool.net.cn>=D0= =B4=B5=C0=A3=BA
=D4=DA 2010-06-02=C8=FD=B5=C4 10:37 +0800=A3=AClwl=D0=B4=B5=C0=A3=BA
> is all the 4 servers' MEM  almost 100%?

Yes

> =D4=DA 2010=C4=EA6=D4=C22=C8=D5 =C9=CF=CE=E710:12=A3=ACShuai Yuan <= yuanshuai@supertool.net.cn>=D0=B4
> =B5=C0=A3=BA
>         Thanks lwl.
>
>         Then is there anyway of tuning this, faste= r flush to disk or
>         else?
>
>         Cheers,
>
>         Kevin
>
>         =D4=DA 2010-06-02=C8=FD=B5=C4 09:57 +0800= =A3=AClwl=D0=B4=B5=C0=A3=BA
>
>         > MEM: almost 100% (16GB)
>         > -----------------
>         > maybe this is the bottleneck.
>         > writing concerns Memtable and SSTable= in memory.
>         >
>         > =D4=DA 2010=C4=EA6=D4=C22=C8=D5 =C9= =CF=CE=E79:48=A3=ACShuai Yuan
>         <yuanshuai@supertool.net.cn>=D0=B4
>         > =B5=C0=A3=BA
>         >         =D4=DA 20= 10-06-01=B6=FE=B5=C4 15:00 -0500=A3=ACJonathan Shook=D0=B4=B5=C0=A3=BA
>         >         > Also= , what are you meaning specifically by 'slow'?
>         Which
>         >         measureme= nts
>         >         > are = you looking at. What are your baseline
>         constraints for
>         >         your test=
>         >         > syst= em?
>         >         >
>         >
>         >         Actually,= the problem is the utilizaton of
>         resources(for a
>         >         single >         >         machine):=
>         >         CPU: 700%= / 1600% (16 cores)
>         >         MEM: almo= st 100% (16GB)
>         >         Swap: alm= ost 0%
>         >         Disk IO(w= rite): 20~30MB / 200MB (7.2k raid5,
>         benchmarked
>         >         previousl= y)
>         >         NET: up t= o 100Mbps / 950Mbps (1Gbps, tuned and
>         benchmarked
>         >         previousl= y)
>         >
>         >         So the sp= eed of generating load, about 15M/s as
>         reported
>         >         before se= ems
>         >         quite slo= w to me. I assume the system should get at
>         least
>         >         about 50M= B/s
>         >         of Disk I= O speed.
>         >
>         >         MEM? I do= n't think it plays a major role in this
>         writing game.
>         >         What'= s
>         >         the bottl= eneck of the system?
>         >
>         >         P.S
>         >         about Con= sistency Level, I've tried ONE/DCQUORUM and
>         found ONE
>         >         is about<= br> >         >         10-15% fa= ster. However that's neither a promising
>         result.
>         >
>         >         Thanks! >         >
>         >         Kevin
>         >
>         >         >
>         >         > 2010= /6/1 =CA=B7=D3=A2=BD=DC <shi= yingjie1983@gmail.com>:
>         >         > >= Hi, It would be better if we know which
>         Consistency Level
>         >         did you c= hoose,
>         >         > >= and what is the schema of test data?
>         >         > >=
>         >         > >= =D4=DA 2010=C4=EA6=D4=C21=C8=D5 =CF=C2=CE=E74:48=A3=ACShuai Yuan
>         >         <yuanshuai@supertool.net.cn>= =D0=B4=B5=C0=A3=BA
>         >         > >= >
>         >         > >= > Hi all,
>         >         > >= >
>         >         > >= > I'm testing writing speed of cassandra with 4
>         servers.
>         >         I'm c= onfused by
>         >         > >= > the behavior of cassandra.
>         >         > >= >
>         >         > >= > ---env---
>         >         > >= > load-data app written in c++, using
>         libcassandra (w/
>         >         modified = batch
>         >         > >= > insert)
>         >         > >= > 20 writing threads in 2 processes running on 2
>         servers
>         >         > >= >
>         >         > >= > ---optimization---
>         >         > >= > 1.turn log level to INFO
>         >         > >= > 2.JVM has 8G heap
>         >         > >= > 3.32 concurrent read & 128 write in
>         storage-conf.xml,
>         >         other cac= he
>         >         > >= > enlarged as well.
>         >         > >= >
>         >         > >= > ---result---
>         >         > >= > 1-monitoring by `date;nodetool -h host ring`
>         >         > >= > I add all load together and measure the writing
>         speed by
>         >         > >= > (load_difference / time_difference), and I get
>         about
>         >         15MB/s fo= r the
>         >         > >= > whole cluster.
>         >         > >= >
>         >         > >= > 2-monitoring by `iostat -m 10`
>         >         > >= > I can watch the disk_io from the system level
>         and have
>         >         about 10M= B/s -
>         >         > >= > 65MB/s for a single machine. Very big variance
>         over time.
>         >         > >= >
>         >         > >= > 3-monitoring by `iptraf -g`
>         >         > >= > In this way I watch the communication between
>         servers and
>         >         get about=
>         >         > >= > 10MB/s for a single machine.
>         >         > >= >
>         >         > >= > ---opinion---
>         >         > >= > So, have you checked the writing speed of
>         cassandra? I
>         >         feel it&#= 39;s quite
>         >         > >= > slow currently.
>         >         > >= >
>         >         > >= > Could anyone confirm this is the normal writing
>         speed of
>         >         cassandra= , or
>         >         > >= > please provide someway of improving it?
>         >         > >= > --
>         >         > >= > Kevin Yuan
>         >         > >= > www.yuan-shua= i.info
>         >         > >= >
>         >         > >= >
>         >         > >=
>         >         > >=
>         >         >
>         >
>         >
>         >         --
>         >
>         >         Kevin Yua= n
>         >         www.yuan-shuai.info >         >
>         >
>         >
>         >
>         >
>
>
>         --
>         Shuai Yuan =D4=AC=CB=A7
>         Supertool Corp. =B1=B1=BE=A9=D1=A7=D6=AE= =CD=BE=CD=F8=C2=E7=BF=C6=BC=BC=D3=D0=CF=DE=B9=AB=CB=BE
>
>         www.yuan-shuai.info
>
>
>

--
Shuai Yuan =D4=AC=CB=A7
Supertool Corp. =B1=B1=BE=A9=D1=A7=D6=AE=CD=BE=CD=F8=C2=E7=BF=C6=BC=BC=D3= =D0=CF=DE=B9=AB=CB=BE
www.yuan-shuai.inf= o



--00c09f8c24ad2ed3f504880394c7--