cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Menon <ra...@apigee.com>
Subject Re: nodetool repair keeping an empty cluster busy
Date Wed, 11 Dec 2013 08:56:47 GMT
Sven

So basically when you run a repair you are essentially telling your cluster
to run a validation compaction, which generates a merkle tree on all the
nodes. These trees are used to identify the inconsistencies. So there is
quite a bit of streaming which you see as your network traffic.

Rahul


On Wed, Dec 11, 2013 at 11:02 AM, Sven Stark <sven.stark@m-square.com.au>wrote:

> Corollary:
>
> what is getting shipped over the wire? The ganglia screenshot shows the
> network traffic on all the three hosts on which I ran the nodetool repair.
>
> [image: Inline image 1]
>
> remember
>
> UN  10.1.2.11  107.47 KB  256     32.9%
>  1f800723-10e4-4dcd-841f-73709a81d432  rack1
> UN  10.1.2.10  127.67 KB  256     32.4%
>  bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639  rack1
> UN  10.1.2.12  107.62 KB  256     34.7%
>  5258f178-b20e-408f-a7bf-b6da2903e026  rack1
>
> Much appreciated.
> Sven
>
>
> On Wed, Dec 11, 2013 at 3:56 PM, Sven Stark <sven.stark@m-square.com.au>wrote:
>
>> Howdy!
>>
>> Not a matter of life or death, just curious.
>>
>> I've just stood up a three node cluster (v1.2.8) on three c3.2xlarge
>> boxes in AWS. Silly me forgot the correct replication factor for one of the
>> needed keyspaces. So I changed it via cli and ran a nodetool repair.
>> Well .. there is no data at all in the keyspace yet, only the definition
>> and nodetool repair ran about 20minutes using 2 of the 8 CPU fully.
>>
>> Any hints what nodetool repair is doing on an empty cluster that makes
>> the host spin so hard?
>>
>> Cheers,
>> Sven
>>
>> ==========================
>>
>> Tasks: 125 total,   1 running, 124 sleeping,   0 stopped,   0 zombie
>> Cpu(s): 22.7%us,  1.0%sy,  2.9%ni, 73.0%id,  0.0%wa,  0.0%hi,  0.4%si,
>>  0.0%st
>> Mem:  15339196k total,  7474360k used,  7864836k free,   251904k buffers
>> Swap:        0k total,        0k used,        0k free,   798324k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 10840 cassandr  20   0 8354m 4.1g  19m S  218 28.0  35:25.73 jsvc
>> 16675 kafka     20   0 3987m 192m  12m S    2  1.3   0:47.89 java
>> 20328 root      20   0 5613m 569m  16m S    2  3.8   1:35.13 jsvc
>>  5969 exhibito  20   0 6423m 116m  12m S    1  0.8   0:25.87 java
>> 14436 tomcat7   20   0 3701m 167m  11m S    1  1.1   0:25.80 java
>>  6278 exhibito  20   0 6487m 119m 9984 S    0  0.8   0:22.63 java
>> 17713 storm     20   0 6033m 159m  11m S    0  1.1   0:10.99 java
>> 18769 storm     20   0 5773m 156m  11m S    0  1.0   0:10.71 java
>>
>> root@xxx-01:~# nodetool -h `hostname` status
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address    Load       Tokens  Owns   Host ID
>>       Rack
>> UN  10.1.2.11  107.47 KB  256     32.9%
>>  1f800723-10e4-4dcd-841f-73709a81d432  rack1
>> UN  10.1.2.10  127.67 KB  256     32.4%
>>  bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639  rack1
>> UN  10.1.2.12  107.62 KB  256     34.7%
>>  5258f178-b20e-408f-a7bf-b6da2903e026  rack1
>>
>> root@xxx-01:~# nodetool -h `hostname` compactionstats
>> pending tasks: 1
>>           compaction type        keyspace   column family       completed
>>           total      unit  progress
>> Active compaction remaining time :        n/a
>>
>> root@xxx-01:~# nodetool -h `hostname` netstats
>> Mode: NORMAL
>> Not sending any streams.
>> Not receiving any streams.
>> Read Repair Statistics:
>> Attempted: 0
>> Mismatch (Blocking): 0
>> Mismatch (Background): 0
>> Pool Name                    Active   Pending      Completed
>> Commands                        n/a         0          57155
>> Responses                       n/a         0          14573
>>
>
>

Mime
View raw message