Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.colby@gmail.com
 designates 209.85.214.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=aPzeBV7ARWdsrK29ob3swHRf5RAYG0DlGJGYuQVWHWSk9Vxu/osA641lE7a0qfMGtO
         tIsOqDOHtKUpoK90us/5xoQqC7J+f494/j/BDyeRX2WiyZej4AhJHsap7iZq6GyX7dqg
         RwmzGDuF2M7KlNsE0Abbjw2Dxda9mLI63f8xY=
MIME-Version: 1.0
In-Reply-To: <BANLkTi=tQrSm5XesfhQQ3EkrrJcmXJYWpA@mail.gmail.com>
References: <BANLkTikhN_iZs6sZWL4nEzBvyEGm_FzSHA@mail.gmail.com>
	<BANLkTi=tQrSm5XesfhQQ3EkrrJcmXJYWpA@mail.gmail.com>
Date: Fri, 27 May 2011 18:06:35 +0200
Message-ID: <BANLkTinbtCPgTtangk6=XVo01yA8tM7QwA@mail.gmail.com>
Subject: Re: average repair/bootstrap durations
From: Jonathan Colby <jonathan.colby@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Ed!   I was thinking about surrendering more memory to mmap
operations.  I'm going to try bringing the Xmx down to 4G

On Fri, May 27, 2011 at 5:19 PM, Edward Capriolo <edlinuxguru@gmail.com> wr=
ote:
>
>
> On Fri, May 27, 2011 at 9:08 AM, Jonathan Colby <jonathan.colby@gmail.com=
>
> wrote:
>>
>> Hi -
>>
>> Operations =A0like repair and bootstrap on nodes in our cluster (average
>> load 150GB each) take a very long time.
>>
>> By long I mean 1-2 days. =A0 With nodetool "netstats" I can see the
>> progress % very slowly progressing.
>>
>> I guess there are some throttling mechanisms built into cassandra.
>> And yes there is also production load on these nodes so it is somewhat
>> understandable. Also some of out compacted data files are as 50-60 GB
>> each.
>>
>> I was just wondering if these times are similar to what other people
>> are experiencing or if there is a serious configuration problem with
>> our setup.
>>
>> So what have you guys seen with operations like loadbalance,repair,
>> cleanup, bootstrap on nodes with large amounts of data??
>>
>> I'm not seeing too many full garbage collections. =A0Other minor GCs are
>> well under a second.
>>
>> Setup info:
>> 0.7.4
>> 5 GB heap
>> 8 GB =A0ram
>> 64 bit linux os
>> AMD quad core HP blades
>> CMS Garbage collector with default cassandra settings
>> 1 TB raid 0 sata disks
>> across 2 datacenters, but operations within the same dc take very long
>> too.
>>
>>
>> This is a netstat output of a bootstrap that has been going on for 3+
>> hours:
>>
>> Mode: Normal
>> Streaming to: /10.47.108.103
>>
>> /var/lib/cassandra/data/DFS/main-f-1541-Data.db/(0,32842490722),(3284249=
0722,139556639427),(139556639427,161075890783)
>> =A0 =A0 =A0 =A0 progress=3D94624588642/161075890783 - 58%
>> =A0 /var/lib/cassandra/data/DFS/main-f-1455-Data.db/(0,660743002)
>> =A0 =A0 =A0 =A0 progress=3D0/660743002 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1444-Data.db/(0,32816130132),(3281613=
0132,71465138397),(71465138397,90968640033)
>> =A0 =A0 =A0 =A0 progress=3D0/90968640033 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1540-Data.db/(0,931632934),(931632934=
,2621052149),(2621052149,3236107041)
>> =A0 =A0 =A0 =A0 progress=3D0/3236107041 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1488-Data.db/(0,33428780851),(3342878=
0851,110546591227),(110546591227,110851587206)
>> =A0 =A0 =A0 =A0 progress=3D0/110851587206 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1542-Data.db/(0,24091168),(24091168,9=
7485080),(97485080,108233211)
>> =A0 =A0 =A0 =A0 progress=3D0/108233211 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1544-Data.db/(0,3646406),(3646406,180=
65308),(18065308,25776551)
>> =A0 =A0 =A0 =A0 progress=3D0/25776551 - 0%
>> =A0 /var/lib/cassandra/data/DFS/main-f-1452-Data.db/(0,676616940)
>> =A0 =A0 =A0 =A0 progress=3D0/676616940 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1548-Data.db/(0,6957269),(6957269,489=
66550),(48966550,51499779)
>> =A0 =A0 =A0 =A0 progress=3D0/51499779 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1552-Data.db/(0,237153399),(237153399=
,750466875),(750466875,898056853)
>> =A0 =A0 =A0 =A0 progress=3D0/898056853 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1554-Data.db/(0,45155582),(45155582,1=
95640768),(195640768,247592141)
>> =A0 =A0 =A0 =A0 progress=3D0/247592141 - 0%
>> =A0 /var/lib/cassandra/data/DFS/main-f-1449-Data.db/(0,2812483216)
>> =A0 =A0 =A0 =A0 progress=3D0/2812483216 - 0%
>>
>> /var/lib/cassandra/data/DFS/main-f-1545-Data.db/(0,107648943),(107648943=
,434575065),(434575065,436667186)
>> =A0 =A0 =A0 =A0 progress=3D0/436667186 - 0%
>> Not receiving any streams.
>> Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending =A0 =
=A0 =A0Completed
>> Commands =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0n/a =A0 =A0 =A0 =
=A0 0 =A0 =A0 =A0 =A0 134283
>> Responses =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 n/a =A0 =A0 =A0 =
=A0 0 =A0 =A0 =A0 =A0 192438
>
> That is a little long but every case is diffent par. With low requiest lo=
ad
> and some heavy server iron RAID,RAM you can see a compaction move really
> fast 300 GB in 4-6 hours. With enough load one of these operations
> compact,cleanup,join can get really bogged down to the point where it alm=
ost
> does not move. Sometimes that is just the way it is based on how fragment=
ed
> your rows are and how fast your gear is. Not pushing your Cassandra cache=
s
> up to your JVM limit can help. If your heap is often near full you can ha=
ve
> jvm memory fragmentation which slows things down.
>
> 0.8 has some more tuning options for compaction, multi-threaded, knobs fo=
r
> effective rate.
>
> I notice you are using:
> 5 GB heap
> 8 GB =A0ram
>
> So your RAM/DATA ratio is on the lower site. I think unless you have a go=
od
> use case for row cache less XMx is more, but that is a minor tweak.
>