They are showing up as completed? Is this correct:
Pool Name Active Pending Completed
STREAMSTAGE 0 0 0
RESPONSESTAGE 0 0 0
ROWREADSTAGE 0 0 517446
LBOPERATIONS 0 0 0
MESSAGEDESERIALIZERPOOL 0 0 0
GMFD 0 0 0
LBTARGET 0 0 0
CONSISTENCYMANAGER 0 0 0
ROWMUTATIONSTAGE 0 0 1353622
MESSAGESTREAMINGPOOL 0 0 0
LOADBALANCERSTAGE 0 0 0
FLUSHSORTERPOOL 0 0 0
MEMTABLEPOSTFLUSHER 0 0 0
FLUSHWRITERPOOL 0 0 0
AESERVICESTAGE 0 0 0
On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
> you need to figure out where the memory is going. check tpstats, if
> the pending ops are large somewhere that means you're just generating
> insert ops faster than it can handle.
>
> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <sonnyheer@gmail.com> wrote:
>> note: I'm using the Thrift API to insert. The commitLog directory
>> continues to grow. The heap size continues to grow as well.
>>
>> I decreased MemtableSizeInMB size, but noticed no changes. Any idea
>> what is causing this, and/or what property i need to tweek to
>> alleviate this? What is the "insert threshold"?
>>
>> I moved to a more powerful node as well, it still ended up failing
>> just after a longer period.
>>
>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>>>
>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <sonnyheer@gmail.com> wrote:
>>>> Currently running on a single node with intensive write operations.
>>>>
>>>>
>>>> After running for a while...
>>>>
>>>> Client starts outputting:
>>>>
>>>> TimedOutException()
>>>> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
>>>> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
>>>> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>>>>
>>>> Cassandra starts outputting:
>>>>
>>>> INFO 08:08:49,864 Cassandra starting up...
>>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
>>>> 220554976 used; max is 1190723584
>>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
>>>> 230245792 used; max is 1190723584
>>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
>>>> 250127792 used; max is 1190723584
>>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
>>>> 261657720 used; max is 1190723584
>>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
>>>> 273764040 used; max is 1190723584
>>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
>>>> 282912656 used; max is 1190723584
>>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
>>>> 294861824 used; max is 1190723584
>>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
>>>> 304045640 used; max is 1190723584
>>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
>>>> 312964344 used; max is 1190723584
>>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
>>>> 324961320 used; max is 1190723584
>>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
>>>> 345874144 used; max is 1190723584
>>>> INFO 08:20:27,357 Creating new commitlog segment
>>>> /var/lib/cassandra/commitlog/CommitLog1271866827357.log
>>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
>>>> 366587984 used; max is 1190723584
>>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
>>>> 384059832 used; max is 1190723584
>>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
>>>> 383631432 used; max is 1190723584
>>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
>>>> 399697888 used; max is 1190723584
>>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
>>>> 415428952 used; max is 1190723584
>>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
>>>> 433542656 used; max is 1190723584
>>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
>>>> 452030560 used; max is 1190723584
>>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
>>>> 464013992 used; max is 1190723584
>>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
>>>> 471978840 used; max is 1190723584
>>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
>>>> 475912120 used; max is 1190723584
>>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
>>>> 483742344 used; max is 1190723584
>>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
>>>> 491761672 used; max is 1190723584
>>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
>>>> 507442800 used; max is 1190723584
>>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
>>>> 523028304 used; max is 1190723584
>>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
>>>> 530865568 used; max is 1190723584
>>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
>>>> 542690216 used; max is 1190723584
>>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
>>>> 550728232 used; max is 1190723584
>>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
>>>> 573981584 used; max is 1190723584
>>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
>>>> 563891472 used; max is 1190723584
>>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
>>>> 575756040 used; max is 1190723584
>>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
>>>> 583631432 used; max is 1190723584
>>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
>>>> 595405816 used; max is 1190723584
>>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
>>>> 603389432 used; max is 1190723584
>>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
>>>> 615150584 used; max is 1190723584
>>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
>>>> 623125104 used; max is 1190723584
>>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
>>>> 634661008 used; max is 1190723584
>>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
>>>> 646272840 used; max is 1190723584
>>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
>>>> 653978688 used; max is 1190723584
>>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
>>>> 665462760 used; max is 1190723584
>>>> INFO 08:26:44,895 Creating new commitlog segment
>>>> /var/lib/cassandra/commitlog/CommitLog1271867204895.log
>>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
>>>> 673423168 used; max is 1190723584
>>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
>>>> 685277616 used; max is 1190723584
>>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
>>>> 692803112 used; max is 1190723584
>>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
>>>> 700213488 used; max is 1190723584
>>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
>>>> 708002984 used; max is 1190723584
>>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
>>>> 719547672 used; max is 1190723584
>>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
>>>> 727261696 used; max is 1190723584
>>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
>>>> 738760480 used; max is 1190723584
>>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
>>>> 746376392 used; max is 1190723584
>>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
>>>> 754102376 used; max is 1190723584
>>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
>>>> 757926896 used; max is 1190723584
>>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
>>>> 763964048 used; max is 1190723584
>>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
>>>> 771496792 used; max is 1190723584
>>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
>>>> 779088496 used; max is 1190723584
>>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
>>>> 786858536 used; max is 1190723584
>>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
>>>> 794395528 used; max is 1190723584
>>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
>>>> 805862000 used; max is 1190723584
>>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
>>>> 813708392 used; max is 1190723584
>>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
>>>> 821245568 used; max is 1190723584
>>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
>>>> 828614768 used; max is 1190723584
>>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
>>>> 835962544 used; max is 1190723584
>>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
>>>> 844183384 used; max is 1190723584
>>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
>>>> 847924056 used; max is 1190723584
>>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
>>>> 841384040 used; max is 1190723584
>>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
>>>> 839554448 used; max is 1190723584
>>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
>>>> 850851600 used; max is 1190723584
>>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
>>>> 869886032 used; max is 1190723584
>>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
>>>> 881346312 used; max is 1190723584
>>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
>>>> 892677840 used; max is 1190723584
>>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
>>>> 911084096 used; max is 1190723584
>>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
>>>> 922370800 used; max is 1190723584
>>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
>>>> 929916416 used; max is 1190723584
>>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
>>>> 941201152 used; max is 1190723584
>>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
>>>> 948631600 used; max is 1190723584
>>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
>>>> 956424384 used; max is 1190723584
>>>> INFO 08:33:14,864 Creating new commitlog segment
>>>> /var/lib/cassandra/commitlog/CommitLog1271867594864.log
>>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
>>>> 972976288 used; max is 1190723584
>>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
>>>> 981829584 used; max is 1190723584
>>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
>>>> 995907192 used; max is 1190723584
>>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
>>>> 1011914464 used; max is 1190723584
>>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
>>>> reclaimed leaving 1015729984 used; max is 1190723584
>>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
>>>> reclaimed leaving 1019176200 used; max is 1190723584
>>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
>>>> reclaimed leaving 1022673936 used; max is 1190723584
>>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
>>>> reclaimed leaving 1025979696 used; max is 1190723584
>>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
>>>> reclaimed leaving 1028992352 used; max is 1190723584
>>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
>>>> reclaimed leaving 1031961224 used; max is 1190723584
>>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
>>>> reclaimed leaving 1034536576 used; max is 1190723584
>>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
>>>> reclaimed leaving 1037266456 used; max is 1190723584
>>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
>>>> reclaimed leaving 1039709528 used; max is 1190723584
>>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
>>>> reclaimed leaving 1042230248 used; max is 1190723584
>>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
>>>> reclaimed leaving 1044527600 used; max is 1190723584
>>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
>>>> reclaimed leaving 1046621520 used; max is 1190723584
>>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
>>>> reclaimed leaving 1048677040 used; max is 1190723584
>>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
>>>> reclaimed leaving 1050563688 used; max is 1190723584
>>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
>>>> reclaimed leaving 1052423824 used; max is 1190723584
>>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
>>>> reclaimed leaving 1054138992 used; max is 1190723584
>>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
>>>> reclaimed leaving 1055761280 used; max is 1190723584
>>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
>>>> reclaimed leaving 1057201248 used; max is 1190723584
>>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
>>>> reclaimed leaving 1058496000 used; max is 1190723584
>>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
>>>> reclaimed leaving 1060011424 used; max is 1190723584
>>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
>>>> reclaimed leaving 1061388688 used; max is 1190723584
>>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
>>>> reclaimed leaving 1062658344 used; max is 1190723584
>>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
>>>> reclaimed leaving 1063834408 used; max is 1190723584
>>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
>>>> reclaimed leaving 1064800992 used; max is 1190723584
>>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
>>>> reclaimed leaving 1065798256 used; max is 1190723584
>>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
>>>> reclaimed leaving 1066867192 used; max is 1190723584
>>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
>>>> reclaimed leaving 1067925416 used; max is 1190723584
>>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
>>>> reclaimed leaving 1068886912 used; max is 1190723584
>>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
>>>> reclaimed leaving 1069705264 used; max is 1190723584
>>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
>>>> reclaimed leaving 1070568224 used; max is 1190723584
>>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
>>>> reclaimed leaving 1071366040 used; max is 1190723584
>>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
>>>> reclaimed leaving 1072168400 used; max is 1190723584
>>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
>>>> reclaimed leaving 1072743152 used; max is 1190723584
>>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
>>>> reclaimed leaving 1073334720 used; max is 1190723584
>>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
>>>> reclaimed leaving 1073930576 used; max is 1190723584
>>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
>>>> reclaimed leaving 1074548160 used; max is 1190723584
>>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
>>>> reclaimed leaving 1075122368 used; max is 1190723584
>>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
>>>> reclaimed leaving 1075699904 used; max is 1190723584
>>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
>>>> reclaimed leaving 1076227648 used; max is 1190723584
>>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
>>>> reclaimed leaving 1076747648 used; max is 1190723584
>>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
>>>> reclaimed leaving 1077152296 used; max is 1190723584
>>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
>>>> reclaimed leaving 1077567128 used; max is 1190723584
>>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
>>>> reclaimed leaving 1077987824 used; max is 1190723584
>>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
>>>> reclaimed leaving 1078393448 used; max is 1190723584
>>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
>>>> reclaimed leaving 1078708784 used; max is 1190723584
>>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
>>>> reclaimed leaving 1079074464 used; max is 1190723584
>>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
>>>> reclaimed leaving 1079409016 used; max is 1190723584
>>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
>>>> reclaimed leaving 1079749672 used; max is 1190723584
>>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
>>>> reclaimed leaving 1080014112 used; max is 1190723584
>>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
>>>> reclaimed leaving 1080278992 used; max is 1190723584
>>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
>>>> reclaimed leaving 1080574456 used; max is 1190723584
>>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
>>>> reclaimed leaving 1080884096 used; max is 1190723584
>>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
>>>> reclaimed leaving 1081076888 used; max is 1190723584
>>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
>>>> reclaimed leaving 1081313872 used; max is 1190723584
>>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
>>>> reclaimed leaving 1081528880 used; max is 1190723584
>>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
>>>> reclaimed leaving 1081704648 used; max is 1190723584
>>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
>>>> reclaimed leaving 1081889512 used; max is 1190723584
>>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
>>>> reclaimed leaving 1082072568 used; max is 1190723584
>>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
>>>> reclaimed leaving 1082270680 used; max is 1190723584
>>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
>>>> reclaimed leaving 1082522872 used; max is 1190723584
>>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
>>>> reclaimed leaving 1082632368 used; max is 1190723584
>>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
>>>> reclaimed leaving 1082794472 used; max is 1190723584
>>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
>>>> reclaimed leaving 1082963008 used; max is 1190723584
>>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
>>>> reclaimed leaving 1083117048 used; max is 1190723584
>>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
>>>> reclaimed leaving 1083213304 used; max is 1190723584
>>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
>>>> reclaimed leaving 1083332280 used; max is 1190723584
>>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
>>>> reclaimed leaving 1083448744 used; max is 1190723584
>>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
>>>> reclaimed leaving 1083601864 used; max is 1190723584
>>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
>>>> reclaimed leaving 1083680168 used; max is 1190723584
>>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
>>>> reclaimed leaving 1083814848 used; max is 1190723584
>>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
>>>> reclaimed leaving 1083871152 used; max is 1190723584
>>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
>>>> reclaimed leaving 1083984728 used; max is 1190723584
>>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
>>>> reclaimed leaving 1084063504 used; max is 1190723584
>>>>
>>>
>>
>
