Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 10883 invoked from network); 17 Mar 2011 10:45:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 17 Mar 2011 10:45:43 -0000 Received: (qmail 41590 invoked by uid 500); 17 Mar 2011 10:45:41 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 41566 invoked by uid 500); 17 Mar 2011 10:45:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 41558 invoked by uid 99); 17 Mar 2011 10:45:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 10:45:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ruslan.usifov@gmail.com designates 209.85.210.44 as permitted sender) Received: from [209.85.210.44] (HELO mail-pz0-f44.google.com) (209.85.210.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 10:45:32 +0000 Received: by pzk30 with SMTP id 30so603132pzk.31 for ; Thu, 17 Mar 2011 03:45:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=sHB06zdz6NheZtJ2tXDsLvbJzpiYN00yddYfXjqit9M=; b=gZlgfvBxZ3mmHESvSe8JSY7ZmOkU9bkhBnB6fV6N36xUgg9dqD5WORL4Q1eNAq2N0F QUnr6HXVkIW/NyOi6zQKJLA1IUgjkooYuKf0Ms4EeB50Reof5alQy/Y1GvDEk2CrVGwm Po9DW/cXFGz3f20RK3NGazAQG8tDjHp2Dj5I8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=TgHEo/SH63+H1duZNxVdKymyTY5GglYrQeWnzvaag13WF+yjYQrT7IEx9XCRz6SuZS gMqavmpUVznZz8qYQaOA7/jtpaXW4D95JWWQZqONJGism6L3PAy+L0K3peBCQChKDKaP 8cwAEA7QwkBJfgpN6YUjL32n14jlL+phwhgGk= MIME-Version: 1.0 Received: by 10.143.27.17 with SMTP id e17mr840524wfj.297.1300358710381; Thu, 17 Mar 2011 03:45:10 -0700 (PDT) Received: by 10.143.7.10 with HTTP; Thu, 17 Mar 2011 03:45:10 -0700 (PDT) Date: Thu, 17 Mar 2011 13:45:10 +0300 Message-ID: Subject: Pauses of GC From: ruslan usifov To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=00504502c5c3034c45049eab5bde X-Virus-Checked: Checked by ClamAV on apache.org --00504502c5c3034c45049eab5bde Content-Type: text/plain; charset=ISO-8859-1 Hello Some times i have very long GC pauses: Total time for which application threads were stopped: 0.0303150 seconds 2011-03-17T13:19:56.476+0300: 33295.671: [GC 33295.671: [ParNew: 678855K->20708K(737280K), 0.0271230 secs] 1457643K->806795K(4112384K), 0.027305 0 secs] [Times: user=0.33 sys=0.00, real=0.03 secs] Total time for which application threads were stopped: 0.0291820 seconds 2011-03-17T13:20:32.962+0300: 33332.157: [GC 33332.157: [ParNew: 676068K->23527K(737280K), 0.0302180 secs] 1462155K->817599K(4112384K), 0.030402 0 secs] [Times: user=0.31 sys=0.00, real=0.03 secs] Total time for which application threads were stopped: 0.1270270 seconds 2011-03-17T13:21:11.908+0300: 33371.103: [GC 33371.103: [ParNew: 678887K->21564K(737280K), 0.0268160 secs] 1472959K->823191K(4112384K), 0.027011 0 secs] [Times: user=0.28 sys=0.00, real=0.03 secs] Total time for which application threads were stopped: 0.0293330 seconds 2011-03-17T13:21:50.482+0300: 33409.677: [GC 33409.677: [ParNew: 676924K->21115K(737280K), 0.0281720 secs] 1478551K->829900K(4112384K), 0.028363 0 secs] [Times: user=0.27 sys=0.00, real=0.03 secs] Total time for which application threads were stopped: 0.0339610 seconds 2011-03-17T13:22:32.849+0300: 33452.044: [GC 33452.044: [ParNew: 676475K->25948K(737280K), 0.0317600 secs] 1485260K->842061K(4112384K), 0.031952 0 secs] [Times: user=0.22 sys=0.00, real=0.03 secs] Total time for which application threads were stopped: 0.0344430 seconds 2011-03-17T13:23:14.924+0300: 33494.119: [GC 33494.119: [ParNew: 681308K->25087K(737280K), 0.0282600 secs] 1497421K->848300K(4112384K), 0.028436 0 secs] [Times: user=0.32 sys=0.00, real=0.03 secs] Total time for which application threads were stopped: 0.0309160 seconds 2011-03-17T13:23:57.192+0300: 33536.387: [GC 33536.387: [ParNew: 680447K->24805K(737280K), 0.0299910 secs] 1503660K->855829K(4112384K), 0.030167 0 secs] [Times: user=0.29 sys=0.01, real=0.03 secs] Total time for which application threads were stopped: 0.0324200 seconds 2011-03-17T13:24:01.553+0300: 33540.748: [GC 33540.749: [ParNew: 680165K->31886K(737280K), 0.0495620 secs] 1511189K->936503K(4112384K), 0.049742 0 secs] [Times: user=0.57 sys=0.00, real=0.05 secs] Total time for which application threads were stopped: 0.0507030 seconds 2011-03-17T13:37:56.009+0300: 34375.204: [GC 34375.204: [ParNew: 687246K->28727K(737280K), 0.0244720 secs] 1591863K->942459K(4112384K), 0.024690 0 secs] [Times: user=0.18 sys=0.00, real=0.02 secs] Total time for which application threads were stopped: 806.7442720 seconds Total time for which application threads were stopped: 0.0006590 seconds Total time for which application threads were stopped: 0.0004360 seconds Total time for which application threads were stopped: 0.0004630 seconds Total time for which application threads were stopped: 0.0008120 seconds 2011-03-17T13:37:59.018+0300: 34378.213: [GC 34378.213: [ParNew: 676678K->21640K(737280K), 0.0137740 secs] 1590410K->949991K(4112384K), 0.013961 0 secs] [Times: user=0.13 sys=0.02, real=0.01 secs] Total time for which application threads were stopped: 0.0145920 seconds Total time for which application threads were stopped: 0.1036080 seconds Total time for which application threads were stopped: 0.0585600 seconds Total time for which application threads were stopped: 0.0600550 seconds Total time for which application threads were stopped: 0.0008560 seconds Total time for which application threads were stopped: 0.0006770 seconds Total time for which application threads were stopped: 0.0005910 seconds Total time for which application threads were stopped: 0.0351330 seconds Total time for which application threads were stopped: 0.0329020 seconds Total time for which application threads were stopped: 0.0728490 seconds Total time for which application threads were stopped: 0.0480990 seconds Total time for which application threads were stopped: 0.0804250 seconds 2011-03-17T13:38:04.394+0300: 34383.589: [GC 34383.589: [ParNew: 677000K->8375K(737280K), 0.0218310 secs] 1605351K->944271K(4112384K), 0.0220300 secs] I have follow nodetoll cfstats on hung node: Keyspace: fishdom_tuenti Read Count: 4970999 Read Latency: 1.0267005945887335 ms. Write Count: 1441619 Write Latency: 0.013146585887117193 ms. Pending Tasks: 0 Column Family: decor SSTable count: 3 Space used (live): 1296203532 Space used (total): 1302520037 Memtable Columns Count: 1066 Memtable Data Size: 121742 Memtable Switch Count: 11 Read Count: 108125 Read Latency: 2.809 ms. Write Count: 11261 Write Latency: 0.006 ms. Pending Tasks: 0 Key cache capacity: 300000 Key cache size: 46470 Key cache hit rate: 0.40384615384615385 Row cache: disabled Compacted row minimum size: 36 Compacted row maximum size: 73457 Compacted row mean size: 958 Column Family: adopt SSTable count: 1 Space used (live): 3285283 Space used (total): 6585649 Memtable Columns Count: 14 Memtable Data Size: 322 Memtable Switch Count: 19 Read Count: 208 Read Latency: 0.009 ms. Write Count: 248 Write Latency: 0.005 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 171 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 103 Compacted row mean size: 103 Column Family: user_store SSTable count: 0 Space used (live): 0 Space used (total): 0 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 Column Family: gifts SSTable count: 1 Space used (live): 8767135 Space used (total): 17581781 Memtable Columns Count: 9 Memtable Data Size: 5993 Memtable Switch Count: 11 Read Count: 230 Read Latency: NaN ms. Write Count: 306 Write Latency: 0.004 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 186 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 125 Compacted row maximum size: 14237 Compacted row mean size: 498 Column Family: user_awards SSTable count: 4 Space used (live): 285142885 Space used (total): 285142885 Memtable Columns Count: 399 Memtable Data Size: 21265 Memtable Switch Count: 11 Read Count: 602884 Read Latency: 0.075 ms. Write Count: 3741 Write Latency: 0.003 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 9692 Key cache hit rate: 0.9929742388758782 Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 770 Compacted row mean size: 262 Column Family: aquarium SSTable count: 4 Space used (live): 945412589 Space used (total): 970196634 Memtable Columns Count: 10973 Memtable Data Size: 1805059 Memtable Switch Count: 11 Read Count: 737198 Read Latency: 0.419 ms. Write Count: 259785 Write Latency: 0.011 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 68960 Key cache hit rate: 0.9497041420118343 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 1916 Compacted row mean size: 539 Column Family: user_messages SSTable count: 1 Space used (live): 4648148 Space used (total): 9311688 Memtable Columns Count: 23 Memtable Data Size: 2525 Memtable Switch Count: 19 Read Count: 8782 Read Latency: 0.012 ms. Write Count: 246 Write Latency: 0.012 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 2707 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 180 Compacted row maximum size: 9887 Compacted row mean size: 380 Column Family: user_stat SSTable count: 4 Space used (live): 872814042 Space used (total): 872814042 Memtable Columns Count: 36852 Memtable Data Size: 902102 Memtable Switch Count: 11 Read Count: 1663028 Read Latency: 0.446 ms. Write Count: 406925 Write Latency: 0.010 ms. Pending Tasks: 0 Key cache capacity: 400000 Key cache size: 78611 Key cache hit rate: 0.9583333333333334 Row cache: disabled Compacted row minimum size: 51 Compacted row maximum size: 770 Compacted row mean size: 516 Column Family: user_stat_friend SSTable count: 4 Space used (live): 614446488 Space used (total): 614446488 Memtable Columns Count: 9288 Memtable Data Size: 250169 Memtable Switch Count: 11 Read Count: 237287 Read Latency: 1.147 ms. Write Count: 116691 Write Latency: 0.006 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 39410 Key cache hit rate: 0.8387096774193549 Row cache: disabled Compacted row minimum size: 61 Compacted row maximum size: 258 Compacted row mean size: 199 Column Family: aquarium_friend SSTable count: 4 Space used (live): 7135867112 Space used (total): 7145863363 Memtable Columns Count: 6061 Memtable Data Size: 618699 Memtable Switch Count: 11 Read Count: 333994 Read Latency: 1.358 ms. Write Count: 168818 Write Latency: 0.010 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 47973 Key cache hit rate: 0.8808664259927798 Row cache: disabled Compacted row minimum size: 104 Compacted row maximum size: 924 Compacted row mean size: 304 Column Family: aquarium_messages SSTable count: 1 Space used (live): 5949016 Space used (total): 11912616 Memtable Columns Count: 11 Memtable Data Size: 1355 Memtable Switch Count: 19 Read Count: 26441 Read Latency: 0.010 ms. Write Count: 279 Write Latency: 0.003 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 4095 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 180 Compacted row maximum size: 14237 Compacted row mean size: 382 Column Family: user_params SSTable count: 0 Space used (live): 0 Space used (total): 0 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 0 Read Latency: NaN ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 Column Family: gems SSTable count: 4 Space used (live): 582170315 Space used (total): 582170315 Memtable Columns Count: 8713 Memtable Data Size: 285047 Memtable Switch Count: 11 Read Count: 97307 Read Latency: 0.733 ms. Write Count: 35400 Write Latency: 0.009 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 21120 Key cache hit rate: 0.8691588785046729 Row cache: disabled Compacted row minimum size: 36 Compacted row maximum size: 2759 Compacted row mean size: 691 Column Family: adopt_accept SSTable count: 1 Space used (live): 3531249 Space used (total): 7077144 Memtable Columns Count: 15 Memtable Data Size: 433 Memtable Switch Count: 19 Read Count: 271 Read Latency: 0.011 ms. Write Count: 249 Write Latency: 0.004 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 325 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 179 Compacted row mean size: 109 Column Family: gems_friend SSTable count: 5 Space used (live): 1620588533 Space used (total): 1620588533 Memtable Columns Count: 21441 Memtable Data Size: 577929 Memtable Switch Count: 11 Read Count: 265787 Read Latency: 0.974 ms. Write Count: 135310 Write Latency: 0.011 ms. Pending Tasks: 0 Key cache capacity: 200000 Key cache size: 49702 Key cache hit rate: 0.8641304347826086 Row cache: disabled Compacted row minimum size: 36 Compacted row maximum size: 1916 Compacted row mean size: 536 Column Family: fish SSTable count: 5 Space used (live): 3424896182 Space used (total): 3543066763 Memtable Columns Count: 88706 Memtable Data Size: 10660109 Memtable Switch Count: 11 Read Count: 889507 Read Latency: 0.718 ms. Write Count: 302385 Write Latency: 0.024 ms. Pending Tasks: 0 Key cache capacity: 400000 Key cache size: 90802 Key cache hit rate: 0.9509594882729211 Row cache: disabled Compacted row minimum size: 51 Compacted row maximum size: 51012 Compacted row mean size: 1890 ---------------- --00504502c5c3034c45049eab5bde Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello

Some times i have very long GC pauses:


Total time f= or which application threads were stopped: 0.0303150 seconds
2011-03-17T= 13:19:56.476+0300: 33295.671: [GC 33295.671: [ParNew: 678855K->20708K(73= 7280K), 0.0271230 secs] 1457643K->806795K(4112384K), 0.027305
0 secs] [Times: user=3D0.33 sys=3D0.00, real=3D0.03 secs]
Total time for= which application threads were stopped: 0.0291820 seconds
2011-03-17T13= :20:32.962+0300: 33332.157: [GC 33332.157: [ParNew: 676068K->23527K(7372= 80K), 0.0302180 secs] 1462155K->817599K(4112384K), 0.030402
0 secs] [Times: user=3D0.31 sys=3D0.00, real=3D0.03 secs]
Total time for= which application threads were stopped: 0.1270270 seconds
2011-03-17T13= :21:11.908+0300: 33371.103: [GC 33371.103: [ParNew: 678887K->21564K(7372= 80K), 0.0268160 secs] 1472959K->823191K(4112384K), 0.027011
0 secs] [Times: user=3D0.28 sys=3D0.00, real=3D0.03 secs]
Total time for= which application threads were stopped: 0.0293330 seconds
2011-03-17T13= :21:50.482+0300: 33409.677: [GC 33409.677: [ParNew: 676924K->21115K(7372= 80K), 0.0281720 secs] 1478551K->829900K(4112384K), 0.028363
0 secs] [Times: user=3D0.27 sys=3D0.00, real=3D0.03 secs]
Total time for= which application threads were stopped: 0.0339610 seconds
2011-03-17T13= :22:32.849+0300: 33452.044: [GC 33452.044: [ParNew: 676475K->25948K(7372= 80K), 0.0317600 secs] 1485260K->842061K(4112384K), 0.031952
0 secs] [Times: user=3D0.22 sys=3D0.00, real=3D0.03 secs]
Total time for= which application threads were stopped: 0.0344430 seconds
2011-03-17T13= :23:14.924+0300: 33494.119: [GC 33494.119: [ParNew: 681308K->25087K(7372= 80K), 0.0282600 secs] 1497421K->848300K(4112384K), 0.028436
0 secs] [Times: user=3D0.32 sys=3D0.00, real=3D0.03 secs]
Total time for= which application threads were stopped: 0.0309160 seconds
2011-03-17T13= :23:57.192+0300: 33536.387: [GC 33536.387: [ParNew: 680447K->24805K(7372= 80K), 0.0299910 secs] 1503660K->855829K(4112384K), 0.030167
0 secs] [Times: user=3D0.29 sys=3D0.01, real=3D0.03 secs]
Total time for= which application threads were stopped: 0.0324200 seconds
2011-03-17T13= :24:01.553+0300: 33540.748: [GC 33540.749: [ParNew: 680165K->31886K(7372= 80K), 0.0495620 secs] 1511189K->936503K(4112384K), 0.049742
0 secs] [Times: user=3D0.57 sys=3D0.00, real=3D0.05 secs]
Total time for= which application threads were stopped: 0.0507030 seconds
2011-03-17T13= :37:56.009+0300: 34375.204: [GC 34375.204: [ParNew: 687246K->28727K(7372= 80K), 0.0244720 secs] 1591863K->942459K(4112384K), 0.024690
0 secs] [Times: user=3D0.18 sys=3D0.00, real=3D0.02 secs]
Total time for= which application threads were stopped: 806.7442720 seconds
Total time = for which application threads were stopped: 0.0006590 seconds
Total time= for which application threads were stopped: 0.0004360 seconds
Total time for which application threads were stopped: 0.0004630 secondsTotal time for which application threads were stopped: 0.0008120 seconds2011-03-17T13:37:59.018+0300: 34378.213: [GC 34378.213: [ParNew: 676678K-= >21640K(737280K), 0.0137740 secs] 1590410K->949991K(4112384K), 0.0139= 61
0 secs] [Times: user=3D0.13 sys=3D0.02, real=3D0.01 secs]
Total time for= which application threads were stopped: 0.0145920 seconds
Total time fo= r which application threads were stopped: 0.1036080 seconds
Total time f= or which application threads were stopped: 0.0585600 seconds
Total time for which application threads were stopped: 0.0600550 secondsTotal time for which application threads were stopped: 0.0008560 secondsTotal time for which application threads were stopped: 0.0006770 seconds<= br> Total time for which application threads were stopped: 0.0005910 secondsTotal time for which application threads were stopped: 0.0351330 secondsTotal time for which application threads were stopped: 0.0329020 seconds<= br> Total time for which application threads were stopped: 0.0728490 secondsTotal time for which application threads were stopped: 0.0480990 secondsTotal time for which application threads were stopped: 0.0804250 seconds<= br> 2011-03-17T13:38:04.394+0300: 34383.589: [GC 34383.589: [ParNew: 677000K-&g= t;8375K(737280K), 0.0218310 secs] 1605351K->944271K(4112384K), 0.0220300=
=A0secs]




I have follow nodetoll cfstats on hung node= :

Keyspace: fishdom_tuenti
=A0=A0=A0 Read Count: 4970999
=A0=A0=A0 = Read Latency: 1.0267005945887335 ms.
=A0=A0=A0 Write Count: 1441619
= =A0=A0=A0 Write Latency: 0.013146585887117193 ms.
=A0=A0=A0 Pending Task= s: 0
=A0=A0=A0 =A0=A0=A0 Column Family: decor
=A0=A0=A0 =A0=A0=A0 SSTable count: 3
=A0=A0=A0 =A0=A0=A0 Space used (liv= e): 1296203532
=A0=A0=A0 =A0=A0=A0 Space used (total): 1302520037
=A0= =A0=A0 =A0=A0=A0 Memtable Columns Count: 1066
=A0=A0=A0 =A0=A0=A0 Memtab= le Data Size: 121742
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 11
=A0=A0=A0 =A0=A0=A0 Read Count: 108125
=A0=A0=A0 =A0=A0=A0 Read Latency:= 2.809 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 11261
=A0=A0=A0 =A0=A0=A0= Write Latency: 0.006 ms.
=A0=A0=A0 =A0=A0=A0 Pending Tasks: 0
=A0=A0= =A0 =A0=A0=A0 Key cache capacity: 300000
=A0=A0=A0 =A0=A0=A0 Key cache s= ize: 46470
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: 0.40384615384615385
=A0=A0=A0 = =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compacted row minimum = size: 36
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 73457
=A0=A0= =A0 =A0=A0=A0 Compacted row mean size: 958

=A0=A0=A0 =A0=A0=A0 Colum= n Family: adopt
=A0=A0=A0 =A0=A0=A0 SSTable count: 1
=A0=A0=A0 =A0=A0=A0 Space used (liv= e): 3285283
=A0=A0=A0 =A0=A0=A0 Space used (total): 6585649
=A0=A0=A0= =A0=A0=A0 Memtable Columns Count: 14
=A0=A0=A0 =A0=A0=A0 Memtable Data = Size: 322
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 19
=A0=A0=A0 =A0= =A0=A0 Read Count: 208
=A0=A0=A0 =A0=A0=A0 Read Latency: 0.009 ms.
=A0=A0=A0 =A0=A0=A0 Write Co= unt: 248
=A0=A0=A0 =A0=A0=A0 Write Latency: 0.005 ms.
=A0=A0=A0 =A0= =A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 200000=A0=A0=A0 =A0=A0=A0 Key cache size: 171
=A0=A0=A0 =A0=A0=A0 Key cache = hit rate: NaN
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compacted ro= w minimum size: 87
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 103=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 103

=A0=A0=A0 =A0=A0= =A0 Column Family: user_store
=A0=A0=A0 =A0=A0=A0 SSTable count: 0
=A0=A0=A0 =A0=A0=A0 Space used (live): 0
=A0=A0=A0 =A0=A0=A0 Space used = (total): 0
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count: 0
=A0=A0=A0 = =A0=A0=A0 Memtable Data Size: 0
=A0=A0=A0 =A0=A0=A0 Memtable Switch Coun= t: 0
=A0=A0=A0 =A0=A0=A0 Read Count: 0
=A0=A0=A0 =A0=A0=A0 Read Laten= cy: NaN ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 0
=A0=A0=A0 =A0=A0=A0 Write Latency: Na= N ms.
=A0=A0=A0 =A0=A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key ca= che capacity: 200000
=A0=A0=A0 =A0=A0=A0 Key cache size: 0
=A0=A0=A0 = =A0=A0=A0 Key cache hit rate: NaN
=A0=A0=A0 =A0=A0=A0 Row cache: disable= d
=A0=A0=A0 =A0=A0=A0 Compacted row minimum size: 0
=A0=A0=A0 =A0=A0=A0 Co= mpacted row maximum size: 0
=A0=A0=A0 =A0=A0=A0 Compacted row mean size:= 0

=A0=A0=A0 =A0=A0=A0 Column Family: gifts
=A0=A0=A0 =A0=A0=A0 S= STable count: 1
=A0=A0=A0 =A0=A0=A0 Space used (live): 8767135
=A0=A0=A0 =A0=A0=A0 Space used (total): 17581781
=A0=A0=A0 =A0=A0=A0 Mem= table Columns Count: 9
=A0=A0=A0 =A0=A0=A0 Memtable Data Size: 5993
= =A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 11
=A0=A0=A0 =A0=A0=A0 Read C= ount: 230
=A0=A0=A0 =A0=A0=A0 Read Latency: NaN ms.
=A0=A0=A0 =A0=A0= =A0 Write Count: 306
=A0=A0=A0 =A0=A0=A0 Write Latency: 0.004 ms.
=A0=A0=A0 =A0=A0=A0 Pending= Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 200000
=A0=A0=A0 = =A0=A0=A0 Key cache size: 186
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: Na= N
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compact= ed row minimum size: 125
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 14237
=A0=A0=A0 =A0=A0= =A0 Compacted row mean size: 498

=A0=A0=A0 =A0=A0=A0 Column Family: = user_awards
=A0=A0=A0 =A0=A0=A0 SSTable count: 4
=A0=A0=A0 =A0=A0=A0 = Space used (live): 285142885
=A0=A0=A0 =A0=A0=A0 Space used (total): 285= 142885
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count: 399
=A0=A0=A0 =A0=A0=A0 Memt= able Data Size: 21265
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 11
= =A0=A0=A0 =A0=A0=A0 Read Count: 602884
=A0=A0=A0 =A0=A0=A0 Read Latency:= 0.075 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 3741
=A0=A0=A0 =A0=A0=A0 = Write Latency: 0.003 ms.
=A0=A0=A0 =A0=A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capac= ity: 200000
=A0=A0=A0 =A0=A0=A0 Key cache size: 9692
=A0=A0=A0 =A0=A0= =A0 Key cache hit rate: 0.9929742388758782
=A0=A0=A0 =A0=A0=A0 Row cache= : disabled
=A0=A0=A0 =A0=A0=A0 Compacted row minimum size: 104
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 770
=A0=A0=A0 =A0=A0=A0 = Compacted row mean size: 262

=A0=A0=A0 =A0=A0=A0 Column Family: aqua= rium
=A0=A0=A0 =A0=A0=A0 SSTable count: 4
=A0=A0=A0 =A0=A0=A0 Space u= sed (live): 945412589
=A0=A0=A0 =A0=A0=A0 Space used (total): 970196634<= br> =A0=A0=A0 =A0=A0=A0 Memtable Columns Count: 10973
=A0=A0=A0 =A0=A0=A0 Me= mtable Data Size: 1805059
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 11<= br>=A0=A0=A0 =A0=A0=A0 Read Count: 737198
=A0=A0=A0 =A0=A0=A0 Read Laten= cy: 0.419 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 259785
=A0=A0=A0 =A0= =A0=A0 Write Latency: 0.011 ms.
=A0=A0=A0 =A0=A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capac= ity: 200000
=A0=A0=A0 =A0=A0=A0 Key cache size: 68960
=A0=A0=A0 =A0= =A0=A0 Key cache hit rate: 0.9497041420118343
=A0=A0=A0 =A0=A0=A0 Row ca= che: disabled
=A0=A0=A0 =A0=A0=A0 Compacted row minimum size: 87
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 1916
=A0=A0=A0 =A0=A0=A0= Compacted row mean size: 539

=A0=A0=A0 =A0=A0=A0 Column Family: use= r_messages
=A0=A0=A0 =A0=A0=A0 SSTable count: 1
=A0=A0=A0 =A0=A0=A0 S= pace used (live): 4648148
=A0=A0=A0 =A0=A0=A0 Space used (total): 931168= 8
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count: 23
=A0=A0=A0 =A0=A0=A0 Memta= ble Data Size: 2525
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 19
=A0= =A0=A0 =A0=A0=A0 Read Count: 8782
=A0=A0=A0 =A0=A0=A0 Read Latency: 0.01= 2 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 246
=A0=A0=A0 =A0=A0=A0 Write = Latency: 0.012 ms.
=A0=A0=A0 =A0=A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capac= ity: 200000
=A0=A0=A0 =A0=A0=A0 Key cache size: 2707
=A0=A0=A0 =A0=A0= =A0 Key cache hit rate: NaN
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
= =A0=A0=A0 =A0=A0=A0 Compacted row minimum size: 180
=A0=A0=A0 =A0=A0=A0 = Compacted row maximum size: 9887
=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 380

=A0=A0=A0 =A0=A0=A0= Column Family: user_stat
=A0=A0=A0 =A0=A0=A0 SSTable count: 4
=A0=A0= =A0 =A0=A0=A0 Space used (live): 872814042
=A0=A0=A0 =A0=A0=A0 Space use= d (total): 872814042
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count: 36852 =A0=A0=A0 =A0=A0=A0 Memtable Data Size: 902102
=A0=A0=A0 =A0=A0=A0 Memta= ble Switch Count: 11
=A0=A0=A0 =A0=A0=A0 Read Count: 1663028
=A0=A0= =A0 =A0=A0=A0 Read Latency: 0.446 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 4= 06925
=A0=A0=A0 =A0=A0=A0 Write Latency: 0.010 ms.
=A0=A0=A0 =A0=A0= =A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 400000
=A0=A0=A0 =A0=A0=A0 Key c= ache size: 78611
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: 0.9583333333333= 334
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compa= cted row minimum size: 51
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size= : 770
=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 516

=A0=A0=A0 =A0=A0=A0= Column Family: user_stat_friend
=A0=A0=A0 =A0=A0=A0 SSTable count: 4=A0=A0=A0 =A0=A0=A0 Space used (live): 614446488
=A0=A0=A0 =A0=A0=A0 Sp= ace used (total): 614446488
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count: = 9288
=A0=A0=A0 =A0=A0=A0 Memtable Data Size: 250169
=A0=A0=A0 =A0=A0=A0 Memta= ble Switch Count: 11
=A0=A0=A0 =A0=A0=A0 Read Count: 237287
=A0=A0=A0= =A0=A0=A0 Read Latency: 1.147 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 1166= 91
=A0=A0=A0 =A0=A0=A0 Write Latency: 0.006 ms.
=A0=A0=A0 =A0=A0=A0 P= ending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 200000
=A0=A0=A0 =A0=A0=A0 Key c= ache size: 39410
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: 0.8387096774193= 549
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compa= cted row minimum size: 61
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size= : 258
=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 199

=A0=A0=A0 =A0=A0=A0= Column Family: aquarium_friend
=A0=A0=A0 =A0=A0=A0 SSTable count: 4
= =A0=A0=A0 =A0=A0=A0 Space used (live): 7135867112
=A0=A0=A0 =A0=A0=A0 Sp= ace used (total): 7145863363
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count:= 6061
=A0=A0=A0 =A0=A0=A0 Memtable Data Size: 618699
=A0=A0=A0 =A0=A0=A0 Memta= ble Switch Count: 11
=A0=A0=A0 =A0=A0=A0 Read Count: 333994
=A0=A0=A0= =A0=A0=A0 Read Latency: 1.358 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 1688= 18
=A0=A0=A0 =A0=A0=A0 Write Latency: 0.010 ms.
=A0=A0=A0 =A0=A0=A0 P= ending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 200000
=A0=A0=A0 =A0=A0=A0 Key c= ache size: 47973
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: 0.8808664259927= 798
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compa= cted row minimum size: 104
=A0=A0=A0 =A0=A0=A0 Compacted row maximum siz= e: 924
=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 304

=A0=A0=A0 =A0=A0=A0= Column Family: aquarium_messages
=A0=A0=A0 =A0=A0=A0 SSTable count: 1=A0=A0=A0 =A0=A0=A0 Space used (live): 5949016
=A0=A0=A0 =A0=A0=A0 Spa= ce used (total): 11912616
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count: 11=
=A0=A0=A0 =A0=A0=A0 Memtable Data Size: 1355
=A0=A0=A0 =A0=A0=A0 Memtabl= e Switch Count: 19
=A0=A0=A0 =A0=A0=A0 Read Count: 26441
=A0=A0=A0 = =A0=A0=A0 Read Latency: 0.010 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 279=A0=A0=A0 =A0=A0=A0 Write Latency: 0.003 ms.
=A0=A0=A0 =A0=A0=A0 Pendi= ng Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 200000
=A0=A0=A0 =A0=A0=A0 Key c= ache size: 4095
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: NaN
=A0=A0=A0= =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compacted row minimum= size: 180
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 14237
=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 382

=A0=A0=A0 =A0=A0=A0= Column Family: user_params
=A0=A0=A0 =A0=A0=A0 SSTable count: 0
=A0= =A0=A0 =A0=A0=A0 Space used (live): 0
=A0=A0=A0 =A0=A0=A0 Space used (to= tal): 0
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count: 0
=A0=A0=A0 =A0= =A0=A0 Memtable Data Size: 0
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 0
=A0=A0=A0 =A0=A0=A0 Read Co= unt: 0
=A0=A0=A0 =A0=A0=A0 Read Latency: NaN ms.
=A0=A0=A0 =A0=A0=A0 = Write Count: 0
=A0=A0=A0 =A0=A0=A0 Write Latency: NaN ms.
=A0=A0=A0 = =A0=A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 20000= 0
=A0=A0=A0 =A0=A0=A0 Key cache size: 0
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: NaN
=A0=A0=A0 =A0=A0=A0 Row cach= e: disabled
=A0=A0=A0 =A0=A0=A0 Compacted row minimum size: 0
=A0=A0= =A0 =A0=A0=A0 Compacted row maximum size: 0
=A0=A0=A0 =A0=A0=A0 Compacte= d row mean size: 0

=A0=A0=A0 =A0=A0=A0 Column Family: gems
=A0=A0=A0 =A0=A0=A0 SSTable count: 4
=A0=A0=A0 =A0=A0=A0 Space used (liv= e): 582170315
=A0=A0=A0 =A0=A0=A0 Space used (total): 582170315
=A0= =A0=A0 =A0=A0=A0 Memtable Columns Count: 8713
=A0=A0=A0 =A0=A0=A0 Memtab= le Data Size: 285047
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 11
=A0=A0=A0 =A0=A0=A0 Read Count: 97307
=A0=A0=A0 =A0=A0=A0 Read Latency: = 0.733 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 35400
=A0=A0=A0 =A0=A0=A0 = Write Latency: 0.009 ms.
=A0=A0=A0 =A0=A0=A0 Pending Tasks: 0
=A0=A0= =A0 =A0=A0=A0 Key cache capacity: 200000
=A0=A0=A0 =A0=A0=A0 Key cache s= ize: 21120
=A0=A0=A0 =A0=A0=A0 Key cache hit rate: 0.8691588785046729
=A0=A0=A0 =A0= =A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compacted row minimum siz= e: 36
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 2759
=A0=A0=A0 = =A0=A0=A0 Compacted row mean size: 691

=A0=A0=A0 =A0=A0=A0 Column Fa= mily: adopt_accept
=A0=A0=A0 =A0=A0=A0 SSTable count: 1
=A0=A0=A0 =A0=A0=A0 Space used (liv= e): 3531249
=A0=A0=A0 =A0=A0=A0 Space used (total): 7077144
=A0=A0=A0= =A0=A0=A0 Memtable Columns Count: 15
=A0=A0=A0 =A0=A0=A0 Memtable Data = Size: 433
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 19
=A0=A0=A0 =A0= =A0=A0 Read Count: 271
=A0=A0=A0 =A0=A0=A0 Read Latency: 0.011 ms.
=A0=A0=A0 =A0=A0=A0 Write Co= unt: 249
=A0=A0=A0 =A0=A0=A0 Write Latency: 0.004 ms.
=A0=A0=A0 =A0= =A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 200000=A0=A0=A0 =A0=A0=A0 Key cache size: 325
=A0=A0=A0 =A0=A0=A0 Key cache = hit rate: NaN
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compacted ro= w minimum size: 87
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 179=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 109

=A0=A0=A0 =A0=A0= =A0 Column Family: gems_friend
=A0=A0=A0 =A0=A0=A0 SSTable count: 5
=A0=A0=A0 =A0=A0=A0 Space used (live): 1620588533
=A0=A0=A0 =A0=A0=A0 Sp= ace used (total): 1620588533
=A0=A0=A0 =A0=A0=A0 Memtable Columns Count:= 21441
=A0=A0=A0 =A0=A0=A0 Memtable Data Size: 577929
=A0=A0=A0 =A0= =A0=A0 Memtable Switch Count: 11
=A0=A0=A0 =A0=A0=A0 Read Count: 265787<= br> =A0=A0=A0 =A0=A0=A0 Read Latency: 0.974 ms.
=A0=A0=A0 =A0=A0=A0 Write Co= unt: 135310
=A0=A0=A0 =A0=A0=A0 Write Latency: 0.011 ms.
=A0=A0=A0 = =A0=A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0 Key cache capacity: 20000= 0
=A0=A0=A0 =A0=A0=A0 Key cache size: 49702
=A0=A0=A0 =A0=A0=A0 Key c= ache hit rate: 0.8641304347826086
=A0=A0=A0 =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compacted ro= w minimum size: 36
=A0=A0=A0 =A0=A0=A0 Compacted row maximum size: 1916<= br>=A0=A0=A0 =A0=A0=A0 Compacted row mean size: 536

=A0=A0=A0 =A0=A0= =A0 Column Family: fish
=A0=A0=A0 =A0=A0=A0 SSTable count: 5
=A0=A0= =A0 =A0=A0=A0 Space used (live): 3424896182
=A0=A0=A0 =A0=A0=A0 Space used (total): 3543066763
=A0=A0=A0 =A0=A0=A0 M= emtable Columns Count: 88706
=A0=A0=A0 =A0=A0=A0 Memtable Data Size: 106= 60109
=A0=A0=A0 =A0=A0=A0 Memtable Switch Count: 11
=A0=A0=A0 =A0=A0= =A0 Read Count: 889507
=A0=A0=A0 =A0=A0=A0 Read Latency: 0.718 ms.
=A0=A0=A0 =A0=A0=A0 Write Count: 302385
=A0=A0=A0 =A0=A0=A0 Write Latenc= y: 0.024 ms.
=A0=A0=A0 =A0=A0=A0 Pending Tasks: 0
=A0=A0=A0 =A0=A0=A0= Key cache capacity: 400000
=A0=A0=A0 =A0=A0=A0 Key cache size: 90802=A0=A0=A0 =A0=A0=A0 Key cache hit rate: 0.9509594882729211
=A0=A0=A0 = =A0=A0=A0 Row cache: disabled
=A0=A0=A0 =A0=A0=A0 Compacted row minimum size: 51
=A0=A0=A0 =A0=A0=A0 C= ompacted row maximum size: 51012
=A0=A0=A0 =A0=A0=A0 Compacted row mean = size: 1890

----------------
--00504502c5c3034c45049eab5bde--