cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gianluca Borello <gianl...@draios.com>
Subject Wide rows best practices and GC impact
Date Wed, 03 Dec 2014 01:01:24 GMT
Hi,

We have a cluster (2.0.11) of 6 nodes (RF=3), c3.4xlarge instances, about
50 column families. Cassandra heap takes 8GB out of the 30GB of every
instance.

We mainly store time series-like data, where each data point is a binary
blob of 5-20KB. We use wide rows, and try to put in the same row all the
data that we usually need in a single query (but not more than that). As a
result, our application logic is very simple (since we have to do just one
query to read the data on average) and read/write response times are very
satisfactory. This is a cfhistograms and a cfstats of our heaviest CF:

SSTables per Read
1 sstables: 3198856
2 sstables: 45

Write Latency (microseconds)
      4 us: 37
      5 us: 1247
      6 us: 9987
      7 us: 31442
      8 us: 66121
     10 us: 400503
     12 us: 1158329
     14 us: 2873934
     17 us: 11843616
     20 us: 24464275
     24 us: 30574717
     29 us: 24351624
     35 us: 16788801
     42 us: 3935374
     50 us: 797781
     60 us: 272160
     72 us: 121819
     86 us: 64641
    103 us: 41085
    124 us: 33618
    149 us: 199463
    179 us: 255445
    215 us: 38238
    258 us: 12300
    310 us: 5307
    372 us: 3180
    446 us: 2443
    535 us: 1773
    642 us: 1314
    770 us: 991
    924 us: 748
   1109 us: 606
   1331 us: 465
   1597 us: 433
   1916 us: 453
   2299 us: 484
   2759 us: 983
   3311 us: 976
   3973 us: 338
   4768 us: 312
   5722 us: 237
   6866 us: 198
   8239 us: 163
   9887 us: 138
  11864 us: 115
  14237 us: 231
  17084 us: 550
  20501 us: 603
  24601 us: 635
  29521 us: 875
  35425 us: 731
  42510 us: 497
  51012 us: 476
  61214 us: 347
  73457 us: 331
  88148 us: 273
 105778 us: 143
 126934 us: 92
 152321 us: 47
 182785 us: 16
 219342 us: 5
 263210 us: 2
 315852 us: 2
 379022 us: 1
 454826 us: 1
 545791 us: 1
 654949 us: 0
 785939 us: 0
 943127 us: 1
1131752 us: 1

Read Latency (microseconds)
     20 us: 1
     24 us: 9
     29 us: 18
     35 us: 96
     42 us: 6989
     50 us: 113305
     60 us: 552348
     72 us: 772329
     86 us: 654019
    103 us: 578404
    124 us: 300364
    149 us: 111522
    179 us: 37385
    215 us: 18353
    258 us: 10733
    310 us: 7915
    372 us: 9406
    446 us: 7645
    535 us: 2773
    642 us: 1323
    770 us: 1351
    924 us: 953
   1109 us: 857
   1331 us: 1122
   1597 us: 800
   1916 us: 806
   2299 us: 686
   2759 us: 581
   3311 us: 671
   3973 us: 318
   4768 us: 318
   5722 us: 226
   6866 us: 164
   8239 us: 161
   9887 us: 134
  11864 us: 125
  14237 us: 184
  17084 us: 285
  20501 us: 315
  24601 us: 378
  29521 us: 431
  35425 us: 468
  42510 us: 469
  51012 us: 466
  61214 us: 407
  73457 us: 337
  88148 us: 297
 105778 us: 242
 126934 us: 135
 152321 us: 109
 182785 us: 57
 219342 us: 41
 263210 us: 28
 315852 us: 16
 379022 us: 12
 454826 us: 6
 545791 us: 6
 654949 us: 0
 785939 us: 0
 943127 us: 0
1131752 us: 2

Partition Size (bytes)
    3311 bytes: 1
    3973 bytes: 2
    4768 bytes: 0
    5722 bytes: 2
    6866 bytes: 0
    8239 bytes: 0
    9887 bytes: 2
   11864 bytes: 1
   14237 bytes: 0
   17084 bytes: 0
   20501 bytes: 0
   24601 bytes: 0
   29521 bytes: 3
   35425 bytes: 0
   42510 bytes: 1
   51012 bytes: 1
   61214 bytes: 1
   73457 bytes: 3
   88148 bytes: 1
  105778 bytes: 5
  126934 bytes: 2
  152321 bytes: 4
  182785 bytes: 65
  219342 bytes: 165
  263210 bytes: 268
  315852 bytes: 201
  379022 bytes: 30
  454826 bytes: 248
  545791 bytes: 16
  654949 bytes: 41
  785939 bytes: 259
  943127 bytes: 547
 1131752 bytes: 243
 1358102 bytes: 176
 1629722 bytes: 59
 1955666 bytes: 37
 2346799 bytes: 41
 2816159 bytes: 78
 3379391 bytes: 243
 4055269 bytes: 122
 4866323 bytes: 209
 5839588 bytes: 220
 7007506 bytes: 266
 8409007 bytes: 77
10090808 bytes: 103
12108970 bytes: 1
14530764 bytes: 2
17436917 bytes: 7
20924300 bytes: 410
25109160 bytes: 76

Cell Count per Partition
    3 cells: 5
    4 cells: 0
    5 cells: 0
    6 cells: 2
    7 cells: 0
    8 cells: 0
   10 cells: 2
   12 cells: 1
   14 cells: 0
   17 cells: 0
   20 cells: 1
   24 cells: 3
   29 cells: 1
   35 cells: 1
   42 cells: 0
   50 cells: 0
   60 cells: 3
   72 cells: 0
   86 cells: 1
  103 cells: 0
  124 cells: 11
  149 cells: 3
  179 cells: 4
  215 cells: 10
  258 cells: 13
  310 cells: 2181
  372 cells: 2
  446 cells: 2
  535 cells: 2
  642 cells: 4
  770 cells: 7
  924 cells: 488
 1109 cells: 3
 1331 cells: 24
 1597 cells: 143
 1916 cells: 332
 2299 cells: 2
 2759 cells: 5
 3311 cells: 483
 3973 cells: 0
 4768 cells: 2
 5722 cells: 1
 6866 cells: 1
 8239 cells: 0
 9887 cells: 2
11864 cells: 244
14237 cells: 1
17084 cells: 248
20501 cells: 1
24601 cells: 1
29521 cells: 1
35425 cells: 2
42510 cells: 1
51012 cells: 2
61214 cells: 237


Read Count: 3202919
Read Latency: 0.16807454013042478 ms.
Write Count: 118568574
Write Latency: 0.026566498615391967 ms.
Pending Tasks: 0
  Table: protobuf_by_agent1
  SSTable count: 49
  SSTables in each level: [1, 11/10, 37, 0, 0, 0, 0, 0, 0]
  Space used (live), bytes: 6934395462
  Space used (total), bytes: 6936424794
  SSTable Compression Ratio: 0.17466856216707344
  Number of keys (estimate): 7040
  Memtable cell count: 47412
  Memtable data size, bytes: 171225925
  Memtable switch count: 4164
  Local read count: 3202919
  Local read latency: 0.168 ms
  Local write count: 118568575
  Local write latency: 0.027 ms
  Pending tasks: 0
  Bloom filter false positives: 286
  Bloom filter false ratio: 0.00000
  Bloom filter space used, bytes: 11480
  Compacted partition minimum bytes: 2760
  Compacted partition maximum bytes: 107964792
  Compacted partition mean bytes: 9316821
  Average live cells per slice (last five minutes): 1.0
  Average tombstones per slice (last five minutes): 0.0

(notice how "compacted partition maximum bytes" is 108MB where from
cfhistograms the distribution is much lower than that)

The other day a few nodes went down. The heap was full and full GCs were
running constantly. Tracking down the problem, it looked like we
misconfigured compaction throughput on some nodes, and when pending
compactions went above a certain threshold (~200), the issue started
happening. Normally, heap usage is almost always under 4GB. Restarting the
nodes and temporarily setting compaction throughput to infinite solved the
issue.

I was able to reproduce this issue in a testing environment by just loading
data in the cluster, and setting compaction throughput to a very low value.
Again, when pending compactions went above the threshold, nodes started
going down again. My assumption is that, having rows this wide, compaction
had to bring in memory a lot of data, and this was be worsened when there
were many pending compactions. I did the following experiments:

1) I repeated the same experiment changing the partition key so that each
row would be effectively about 1/10th the size of the rows in the original
scenario. In this case, I didn't experience any node failure, even when
pending compactions went as high as 900.

2) I repeated the same experiment with Cassandra 2.1.2. Even with the
original partition key, nodes never went down. I'm not sure I understand
why, the only thing I can say is that even after a couple hours pending
compactions were never more than 50 (whereas in 2.0.11 it took just about
20 minutes to get to the point of failure)

Do you have some best practices on wide rows sizing? Are we doing something
wrong by using rows this wide or the problem we experienced is unrelated?

Thanks a lot.

Mime
View raw message