Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E3BC1109C2 for ; Wed, 11 Dec 2013 04:56:50 +0000 (UTC) Received: (qmail 3264 invoked by uid 500); 11 Dec 2013 04:56:42 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 3208 invoked by uid 500); 11 Dec 2013 04:56:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 3184 invoked by uid 99); 11 Dec 2013 04:56:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 04:56:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [74.125.82.54] (HELO mail-wg0-f54.google.com) (74.125.82.54) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 04:56:28 +0000 Received: by mail-wg0-f54.google.com with SMTP id n12so5818497wgh.21 for ; Tue, 10 Dec 2013 20:56:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=EEAzIKS3aHFrrOV2KG/3tc8tOsNYmor7CRtwA0anSn4=; b=aOao3gcbQKNdq/rPm67Pfbs+h8XeHABSjU9fqxROcuGCESXzCBKQAg8WxGk774OQfK P/TSZWPpU/Lb6ILzyc7+8TtMYUQ3LuLyawckuAx1hhGuqnbwKUNlyWDWfcSSepfr+QWe /APb9zEuUwmxDWwkSx2B5kNzoY2gO7cp9EwtQYfV5HGCW6ceWskrlUQwe8Ksr26/hPrG 53s6rCJvDMuUylD3TwKAE8cQWC9dDy6WizhNLmVoFbe2dgp+/SG71/P/hyLpNVBFtv8L H+jIUDO+wmQnn8DkbvOZLBj4M9mh15qqjtqdA5BBkB1mweXCJE7k3ca5gI35XZi6/TX0 QJSw== X-Gm-Message-State: ALoCoQlidT9oIdAW3LupyFfaSlD8soUOgwDFaLKTec67bvDp3/edCr6A034iH/vgdMDlD51jwbRV MIME-Version: 1.0 X-Received: by 10.180.187.72 with SMTP id fq8mr22367818wic.26.1386737765402; Tue, 10 Dec 2013 20:56:05 -0800 (PST) Received: by 10.194.151.169 with HTTP; Tue, 10 Dec 2013 20:56:05 -0800 (PST) Date: Wed, 11 Dec 2013 15:56:05 +1100 Message-ID: Subject: nodetool repair keeping an empty cluster busy From: Sven Stark To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c2679ce760a404ed3b0b1e X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2679ce760a404ed3b0b1e Content-Type: text/plain; charset=ISO-8859-1 Howdy! Not a matter of life or death, just curious. I've just stood up a three node cluster (v1.2.8) on three c3.2xlarge boxes in AWS. Silly me forgot the correct replication factor for one of the needed keyspaces. So I changed it via cli and ran a nodetool repair. Well .. there is no data at all in the keyspace yet, only the definition and nodetool repair ran about 20minutes using 2 of the 8 CPU fully. Any hints what nodetool repair is doing on an empty cluster that makes the host spin so hard? Cheers, Sven ========================== Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie Cpu(s): 22.7%us, 1.0%sy, 2.9%ni, 73.0%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 15339196k total, 7474360k used, 7864836k free, 251904k buffers Swap: 0k total, 0k used, 0k free, 798324k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10840 cassandr 20 0 8354m 4.1g 19m S 218 28.0 35:25.73 jsvc 16675 kafka 20 0 3987m 192m 12m S 2 1.3 0:47.89 java 20328 root 20 0 5613m 569m 16m S 2 3.8 1:35.13 jsvc 5969 exhibito 20 0 6423m 116m 12m S 1 0.8 0:25.87 java 14436 tomcat7 20 0 3701m 167m 11m S 1 1.1 0:25.80 java 6278 exhibito 20 0 6487m 119m 9984 S 0 0.8 0:22.63 java 17713 storm 20 0 6033m 159m 11m S 0 1.1 0:10.99 java 18769 storm 20 0 5773m 156m 11m S 0 1.0 0:10.71 java root@xxx-01:~# nodetool -h `hostname` status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.1.2.11 107.47 KB 256 32.9% 1f800723-10e4-4dcd-841f-73709a81d432 rack1 UN 10.1.2.10 127.67 KB 256 32.4% bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639 rack1 UN 10.1.2.12 107.62 KB 256 34.7% 5258f178-b20e-408f-a7bf-b6da2903e026 rack1 root@xxx-01:~# nodetool -h `hostname` compactionstats pending tasks: 1 compaction type keyspace column family completed total unit progress Active compaction remaining time : n/a root@xxx-01:~# nodetool -h `hostname` netstats Mode: NORMAL Not sending any streams. Not receiving any streams. Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool Name Active Pending Completed Commands n/a 0 57155 Responses n/a 0 14573 --001a11c2679ce760a404ed3b0b1e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Howdy!

Not a matter of life or death, just c= urious.=A0

I've just stood up a three node cluster (v1.2.8) on three c3= .2xlarge boxes in AWS. Silly me forgot the correct replication factor for o= ne of the needed keyspaces. So I changed it via cli and ran a nodetool repa= ir.=A0
We= ll .. there is no data at all in the keyspace yet, only the definition and = nodetool repair ran about 20minutes using 2 of the 8 CPU fully.
Any hints what nodetool repair is doing on an empty clu= ster that makes the host spin so hard?
Cheers,
Sven
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=A0

Tasks: 125 total, =A0 1 running, 124 sleeping, = =A0 0 stopped, =A0 0 zombie
Cpu(s): 22.7%us, =A01.0%sy, =A02.9%ni= , 73.0%id, =A00.0%wa, =A00.0%hi, =A00.4%si, =A00.0%st
Mem: =A015339196k total, =A07474360k used, =A07864836k free, =A0 25190= 4k buffers
Swap: =A0 =A0 =A0 =A00k total, =A0 =A0 =A0 =A00k used,= =A0 =A0 =A0 =A00k free, =A0 798324k cached

=A0 PI= D USER =A0 =A0 =A0PR =A0NI =A0VIRT =A0RES =A0SHR S %CPU %MEM =A0 =A0TIME+ = =A0COMMAND
10840 cassandr =A020 =A0 0 8354m 4.1g =A019m S =A0218 28.0 =A035:25.73= jsvc
16675 kafka =A0 =A0 20 =A0 0 3987m 192m =A012m S =A0 =A02 = =A01.3 =A0 0:47.89 java
20328 root =A0 =A0 =A020 =A0 0 5613m 569m= =A016m S =A0 =A02 =A03.8 =A0 1:35.13 jsvc
=A05969 exhibito =A020 =A0 0 6423m 116m =A012m S =A0 =A01 =A00.8 =A0 0= :25.87 java
14436 tomcat7 =A0 20 =A0 0 3701m 167m =A011m S =A0 = =A01 =A01.1 =A0 0:25.80 java
=A06278 exhibito =A020 =A0 0 6487m 1= 19m 9984 S =A0 =A00 =A00.8 =A0 0:22.63 java
17713 storm =A0 =A0 20 =A0 0 6033m 159m =A011m S =A0 =A00 =A01.1 =A0 0= :10.99 java
18769 storm =A0 =A0 20 =A0 0 5773m 156m =A011m S =A0 = =A00 =A01.0 =A0 0:10.71 java

root@xxx-01:~# n= odetool -h `hostname` status
Datacenter: datacenter1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down
|/ S= tate=3DNormal/Leaving/Joining/Moving
-- =A0Address =A0 =A0Load = =A0 =A0 =A0 Tokens =A0Owns =A0 Host ID =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 Rack
UN =A010.1.2.11 =A0107.47 KB =A0256 =A0 =A0 32.9% =A01f800723-10e4-4dc= d-841f-73709a81d432 =A0rack1
UN =A010.1.2.10 =A0127.67 KB =A0256 = =A0 =A0 32.4% =A0bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639 =A0rack1
UN= =A010.1.2.12 =A0107.62 KB =A0256 =A0 =A0 34.7% =A05258f178-b20e-408f-a7bf-= b6da2903e026 =A0rack1

root@xxx-01:~# nodetool -h `hostname` compac= tionstats
pending tasks: 1
=A0 =A0 =A0 =A0 =A0 compacti= on type =A0 =A0 =A0 =A0keyspace =A0 column family =A0 =A0 =A0 completed =A0= =A0 =A0 =A0 =A0 total =A0 =A0 =A0unit =A0progress
Active compaction remaining time : =A0 =A0 =A0 =A0n/a
=
root@xxx-01:~# nodetool -h `hostname` netstats
Mode: NORMAL
Not sending any streams.
Not receiving a= ny streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blo= cking): 0
Mismatch (Background): 0
Pool Name =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending =A0 =A0 =A0Completed
Commands =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0n/a =A0 =A0 = =A0 =A0 0 =A0 =A0 =A0 =A0 =A057155
Responses =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 n/a =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0 =A0 =A014573
--001a11c2679ce760a404ed3b0b1e--