incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sven Stark <sven.st...@m-square.com.au>
Subject Re: nodetool repair keeping an empty cluster busy
Date Wed, 11 Dec 2013 09:35:36 GMT
Hi Rahul,

thanks for replying. Could you please be a bit more specific, though. Eg
what exactly is being compacted - there is/was no data at all in the
cluster save for a few hundred kB in the system CF (see the nodetool status
output). Or - how can those few hundred kB in data generate Gb of network
traffic?

Cheers,
Sven



On Wed, Dec 11, 2013 at 7:56 PM, Rahul Menon <rahul@apigee.com> wrote:

> Sven
>
> So basically when you run a repair you are essentially telling your
> cluster to run a validation compaction, which generates a merkle tree on
> all the nodes. These trees are used to identify the inconsistencies. So
> there is quite a bit of streaming which you see as your network traffic.
>
> Rahul
>
>
> On Wed, Dec 11, 2013 at 11:02 AM, Sven Stark <sven.stark@m-square.com.au>wrote:
>
>> Corollary:
>>
>> what is getting shipped over the wire? The ganglia screenshot shows the
>> network traffic on all the three hosts on which I ran the nodetool repair.
>>
>> [image: Inline image 1]
>>
>> remember
>>
>> UN  10.1.2.11  107.47 KB  256     32.9%
>>  1f800723-10e4-4dcd-841f-73709a81d432  rack1
>> UN  10.1.2.10  127.67 KB  256     32.4%
>>  bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639  rack1
>> UN  10.1.2.12  107.62 KB  256     34.7%
>>  5258f178-b20e-408f-a7bf-b6da2903e026  rack1
>>
>> Much appreciated.
>> Sven
>>
>>
>> On Wed, Dec 11, 2013 at 3:56 PM, Sven Stark <sven.stark@m-square.com.au>wrote:
>>
>>> Howdy!
>>>
>>> Not a matter of life or death, just curious.
>>>
>>> I've just stood up a three node cluster (v1.2.8) on three c3.2xlarge
>>> boxes in AWS. Silly me forgot the correct replication factor for one of the
>>> needed keyspaces. So I changed it via cli and ran a nodetool repair.
>>> Well .. there is no data at all in the keyspace yet, only the definition
>>> and nodetool repair ran about 20minutes using 2 of the 8 CPU fully.
>>>
>>> Any hints what nodetool repair is doing on an empty cluster that makes
>>> the host spin so hard?
>>>
>>> Cheers,
>>> Sven
>>>
>>> ==========================
>>>
>>> Tasks: 125 total,   1 running, 124 sleeping,   0 stopped,   0 zombie
>>> Cpu(s): 22.7%us,  1.0%sy,  2.9%ni, 73.0%id,  0.0%wa,  0.0%hi,  0.4%si,
>>>  0.0%st
>>> Mem:  15339196k total,  7474360k used,  7864836k free,   251904k buffers
>>> Swap:        0k total,        0k used,        0k free,   798324k cached
>>>
>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 10840 cassandr  20   0 8354m 4.1g  19m S  218 28.0  35:25.73 jsvc
>>> 16675 kafka     20   0 3987m 192m  12m S    2  1.3   0:47.89 java
>>> 20328 root      20   0 5613m 569m  16m S    2  3.8   1:35.13 jsvc
>>>  5969 exhibito  20   0 6423m 116m  12m S    1  0.8   0:25.87 java
>>> 14436 tomcat7   20   0 3701m 167m  11m S    1  1.1   0:25.80 java
>>>  6278 exhibito  20   0 6487m 119m 9984 S    0  0.8   0:22.63 java
>>> 17713 storm     20   0 6033m 159m  11m S    0  1.1   0:10.99 java
>>> 18769 storm     20   0 5773m 156m  11m S    0  1.0   0:10.71 java
>>>
>>> root@xxx-01:~# nodetool -h `hostname` status
>>> Datacenter: datacenter1
>>> =======================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address    Load       Tokens  Owns   Host ID
>>>       Rack
>>> UN  10.1.2.11  107.47 KB  256     32.9%
>>>  1f800723-10e4-4dcd-841f-73709a81d432  rack1
>>> UN  10.1.2.10  127.67 KB  256     32.4%
>>>  bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639  rack1
>>> UN  10.1.2.12  107.62 KB  256     34.7%
>>>  5258f178-b20e-408f-a7bf-b6da2903e026  rack1
>>>
>>> root@xxx-01:~# nodetool -h `hostname` compactionstats
>>> pending tasks: 1
>>>           compaction type        keyspace   column family
>>> completed           total      unit  progress
>>> Active compaction remaining time :        n/a
>>>
>>> root@xxx-01:~# nodetool -h `hostname` netstats
>>> Mode: NORMAL
>>> Not sending any streams.
>>> Not receiving any streams.
>>> Read Repair Statistics:
>>> Attempted: 0
>>> Mismatch (Blocking): 0
>>> Mismatch (Background): 0
>>> Pool Name                    Active   Pending      Completed
>>> Commands                        n/a         0          57155
>>> Responses                       n/a         0          14573
>>>
>>
>>
>

Mime
View raw message