cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Unable to repair a node
Date Sun, 14 Aug 2011 10:29:59 GMT
Hello, I've been fighting with my cluster for a couple days now... Running
0.8.1.3, using Hector and loadblancing requests across all nodes.
My question is : how do I get my node back under control so that it runs
like the other two nodes.


It's a 3 node, RF=3 cluster with reads & writes at LC=QUORUM, I only have
counter columns inside super columns. There are 6 keyspaces, each has about
10 column families. I'm using the BOP. Before the sequence of events
described below, I was writing at CL=ALL and reading at CL=ONE. I've
launched repairs multiple times and they have failed for various reasons,
one of them being hitting the limit on number of open files. I've raised it
to 32768 now. I've probably launched repairs when a repair was already
running on the node. At some point compactions were throttled to 16MB / s,
I've removed this limit.

The problem is that  one of my nodes is now impossible to repair (no such
problem with the two others). The load is about 90GB, it should be a
balanced ring but the other nodes are at 60GB. Each repair basically
generates thousands of pending compactions of various types (SSTable build,
minor, major & validation) : it spikes up to 4000 thousands, levels then
spikes up to 8000.Previously, I hit linux limits and had to restart the node
but it doesn't look like the repairs have been improving anything time after
time.
At the same time,

   - the number of SSTables for some keyspaces goes dramatically up (from 3
   or 4 to several dozens).
   - the commit log keeps increasing in size, I'm at 4.3G now, it went up to
   40G when the compaction was throttled at 16MB/s. On the other nodes it's
   around 1GB at most
   - the data directory is bigger than on the other nodes. I've seen it go
   up to 480GB when the compaction was throttled at 16MB/s


Compaction stats:
pending tasks: 5954
          compaction type        keyspace   column family bytes compacted
  bytes total  progress
               ValidationROLLUP_WIFI_COVERAGEPUBLIC_MONTHLY_17
569432689       596621002    95.44%
                    MinorROLLUP_CDMA_COVERAGEPUBLIC_MONTHLY_20
 2751906910      5806164726    47.40%
               ValidationROLLUP_WIFI_COVERAGEPUBLIC_MONTHLY_20
 2570106876      2776508919    92.57%
               ValidationROLLUP_CDMA_COVERAGEPUBLIC_MONTHLY_19
 3010471905      6517183774    46.19%
                    MinorROLLUP_CDMA_COVERAGEPUBLIC_MONTHLY_15
 4132       303015882     0.00%
                    MinorROLLUP_CDMA_COVERAGEPUBLIC_MONTHLY_18
 36302803       595278385     6.10%
                    MinorROLLUP_CDMA_COVERAGEPUBLIC_MONTHLY_17
 24671866        70959088    34.77%
                    MinorROLLUP_CDMA_COVERAGEPUBLIC_MONTHLY_20
 15515781       692029872     2.24%
                    MinorROLLUP_CDMA_COVERAGEPUBLIC_MONTHLY_20
 1607953684      6606774662    24.34%
               ValidationROLLUP_WIFI_COVERAGEPUBLIC_MONTHLY_20
895043380      2776306015    32.24%

My current lsof count for the cassandra user is
root@xxx:/logs/cassandra# lsof -u cassandra| wc -l
13191

What's even weirder is that currently I have 9 compactions running but CPU
is throttled at 1/number of cores half the time (while > 80% the rest of the
time). Could this be because other repairs are happening in the ring ?
Exemple (vmstat 2)
 7  2      0 177632   1596 13868416    0    0  9060    61 5963 5968 40  7 53
 0
 7  0      0 165376   1600 13880012    0    0 41422    28 14027 4608 81 17
 1  0
 8  0      0 159820   1592 13880036    0    0 26830    22 10161 10398 76 19
 4  1
 6  0      0 161792   1592 13882312    0    0 20046    42 7272 4599 81 17  2
 0
 2  0      0 164960   1564 13879108    0    0 17404 26559 6172 3638 79 18  2
 0
 2  0      0 162344   1564 13867888    0    0     6     0 2014 2150 40  2 58
 0
 1  1      0 159864   1572 13867952    0    0     0 41668  958  581 27  0 72
 1
 1  0      0 161972   1572 13867952    0    0     0    89  661  443 17  0 82
 1
 1  0      0 162128   1572 13867952    0    0     0    20  482  398 17  0 83
 0
 2  0      0 162276   1572 13867952    0    0     0   788  485  395 18  0 82
 0
 1  0      0 173896   1572 13867952    0    0     0    29  547  461 17  0 83
 0
 1  0      0 163052   1572 13867920    0    0     0     0  741  620 18  1 81
 0
 1  0      0 162588   1580 13867948    0    0     0    32  523  387 17  0 82
 0
13  0      0 168272   1580 13877140    0    0 12872   269 8056 6725 56  9 34
 0
44  1      0 202536   1612 13835956    0    0 26606   530 7946 3887 79 19  2
 0
48  1      0 406640   1612 13631740    0    0 22006   310 8605 3705 80 18  2
 0
 9  1      0 340300   1620 13697560    0    0 19530   103 8101 3984 84 14  1
 0
 2  0      0 297768   1620 13738036    0    0 12438    10 4115 2628 57  9 34
 0

Thanks

Mime
View raw message