Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 87693 invoked from network); 21 Apr 2010 21:10:36 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 21 Apr 2010 21:10:36 -0000 Received: (qmail 24121 invoked by uid 500); 21 Apr 2010 21:10:35 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 24098 invoked by uid 500); 21 Apr 2010 21:10:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 24090 invoked by uid 99); 21 Apr 2010 21:10:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:10:35 +0000 X-ASF-Spam-Status: No, hits=-0.2 required=10.0 tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-wy0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:10:28 +0000 Received: by wyb35 with SMTP id 35so112437wyb.31 for ; Wed, 21 Apr 2010 14:10:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:received:message-id:subject:to:content-type :content-transfer-encoding; bh=pjniCiEc7JtGQT2/ixgCI51v77KoEvU1IyDgFLlQm5I=; b=ZI7r5QF18UtTh94EVH/VUc7YZap5SIxxttHVgJNHNQoPfKcWbLVmjO8sa5YEf4aayM +NM0+5PhXipM4IUwWWbL2N0YqlKNWc1DEKEhCX0qaDT9/jOV5oY8AeBVfebTkf25vDTK pt8K0HIUYjDfrvudg+aYPANLjxrDGQI0BIRQU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=Yo5J5TcIYWz33PmInuFgMIsWf0qp33HYqkIrpVFLcSnrWkmDAoL7oWUwTsS0bOps0n M8txZOQad9QF5ctVyBfatmvBZu6BMXTJPxd4cfEPU+7os6jOBq12Dpq3nJ385y1+ekMs 6Tzx73VyNbgSx3Tmk3e2ScG+tNx3dkisb5Q/M= MIME-Version: 1.0 Received: by 10.216.17.147 with HTTP; Wed, 21 Apr 2010 14:09:47 -0700 (PDT) In-Reply-To: References: From: Jonathan Ellis Date: Wed, 21 Apr 2010 16:09:47 -0500 Received: by 10.216.172.194 with SMTP id t44mr76552wel.207.1271884207264; Wed, 21 Apr 2010 14:10:07 -0700 (PDT) Message-ID: Subject: Re: Import using cassandra 0.6.1 To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable you need to figure out where the memory is going. check tpstats, if the pending ops are large somewhere that means you're just generating insert ops faster than it can handle. On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer wrote: > note: I'm using the Thrift API to insert. =A0The commitLog directory > continues to grow. =A0The heap size continues to grow as well. > > I decreased MemtableSizeInMB size, but noticed no changes. =A0Any idea > what is causing this, and/or what property i need to tweek to > alleviate this? =A0What is the "insert threshold"? > > I moved to a more powerful node as well, it still ended up failing > just after a longer period. > > On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis wrot= e: >> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts >> >> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer wrote= : >>> Currently running on a single node with intensive write operations. >>> >>> >>> After running for a while... >>> >>> Client starts outputting: >>> >>> TimedOutException() >>> =A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$insert_result.r= ead(Cassandra.java:12232) >>> =A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$Client.recv_ins= ert(Cassandra.java:670) >>> =A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$Client.insert(C= assandra.java:643) >>> >>> Cassandra starts outputting: >>> >>> =A0INFO 08:08:49,864 Cassandra starting up... >>> =A0INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving >>> 220554976 used; max is 1190723584 >>> =A0INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving >>> 230245792 used; max is 1190723584 >>> =A0INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving >>> 250127792 used; max is 1190723584 >>> =A0INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving >>> 261657720 used; max is 1190723584 >>> =A0INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving >>> 273764040 used; max is 1190723584 >>> =A0INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving >>> 282912656 used; max is 1190723584 >>> =A0INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving >>> 294861824 used; max is 1190723584 >>> =A0INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving >>> 304045640 used; max is 1190723584 >>> =A0INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving >>> 312964344 used; max is 1190723584 >>> =A0INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving >>> 324961320 used; max is 1190723584 >>> =A0INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving >>> 345874144 used; max is 1190723584 >>> =A0INFO 08:20:27,357 Creating new commitlog segment >>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log >>> =A0INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving >>> 366587984 used; max is 1190723584 >>> =A0INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving >>> 384059832 used; max is 1190723584 >>> =A0INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving >>> 383631432 used; max is 1190723584 >>> =A0INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving >>> 399697888 used; max is 1190723584 >>> =A0INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving >>> 415428952 used; max is 1190723584 >>> =A0INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving >>> 433542656 used; max is 1190723584 >>> =A0INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving >>> 452030560 used; max is 1190723584 >>> =A0INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving >>> 464013992 used; max is 1190723584 >>> =A0INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving >>> 471978840 used; max is 1190723584 >>> =A0INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving >>> 475912120 used; max is 1190723584 >>> =A0INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving >>> 483742344 used; max is 1190723584 >>> =A0INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving >>> 491761672 used; max is 1190723584 >>> =A0INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving >>> 507442800 used; max is 1190723584 >>> =A0INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving >>> 523028304 used; max is 1190723584 >>> =A0INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving >>> 530865568 used; max is 1190723584 >>> =A0INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving >>> 542690216 used; max is 1190723584 >>> =A0INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving >>> 550728232 used; max is 1190723584 >>> =A0INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving >>> 573981584 used; max is 1190723584 >>> =A0INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving >>> 563891472 used; max is 1190723584 >>> =A0INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving >>> 575756040 used; max is 1190723584 >>> =A0INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving >>> 583631432 used; max is 1190723584 >>> =A0INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving >>> 595405816 used; max is 1190723584 >>> =A0INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving >>> 603389432 used; max is 1190723584 >>> =A0INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving >>> 615150584 used; max is 1190723584 >>> =A0INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving >>> 623125104 used; max is 1190723584 >>> =A0INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving >>> 634661008 used; max is 1190723584 >>> =A0INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving >>> 646272840 used; max is 1190723584 >>> =A0INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving >>> 653978688 used; max is 1190723584 >>> =A0INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving >>> 665462760 used; max is 1190723584 >>> =A0INFO 08:26:44,895 Creating new commitlog segment >>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log >>> =A0INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving >>> 673423168 used; max is 1190723584 >>> =A0INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving >>> 685277616 used; max is 1190723584 >>> =A0INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving >>> 692803112 used; max is 1190723584 >>> =A0INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving >>> 700213488 used; max is 1190723584 >>> =A0INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving >>> 708002984 used; max is 1190723584 >>> =A0INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving >>> 719547672 used; max is 1190723584 >>> =A0INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving >>> 727261696 used; max is 1190723584 >>> =A0INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving >>> 738760480 used; max is 1190723584 >>> =A0INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving >>> 746376392 used; max is 1190723584 >>> =A0INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving >>> 754102376 used; max is 1190723584 >>> =A0INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving >>> 757926896 used; max is 1190723584 >>> =A0INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving >>> 763964048 used; max is 1190723584 >>> =A0INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving >>> 771496792 used; max is 1190723584 >>> =A0INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving >>> 779088496 used; max is 1190723584 >>> =A0INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving >>> 786858536 used; max is 1190723584 >>> =A0INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving >>> 794395528 used; max is 1190723584 >>> =A0INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving >>> 805862000 used; max is 1190723584 >>> =A0INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving >>> 813708392 used; max is 1190723584 >>> =A0INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving >>> 821245568 used; max is 1190723584 >>> =A0INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving >>> 828614768 used; max is 1190723584 >>> =A0INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving >>> 835962544 used; max is 1190723584 >>> =A0INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving >>> 844183384 used; max is 1190723584 >>> =A0INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving >>> 847924056 used; max is 1190723584 >>> =A0INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving >>> 841384040 used; max is 1190723584 >>> =A0INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving >>> 839554448 used; max is 1190723584 >>> =A0INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving >>> 850851600 used; max is 1190723584 >>> =A0INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving >>> 869886032 used; max is 1190723584 >>> =A0INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving >>> 881346312 used; max is 1190723584 >>> =A0INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving >>> 892677840 used; max is 1190723584 >>> =A0INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving >>> 911084096 used; max is 1190723584 >>> =A0INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving >>> 922370800 used; max is 1190723584 >>> =A0INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving >>> 929916416 used; max is 1190723584 >>> =A0INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving >>> 941201152 used; max is 1190723584 >>> =A0INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving >>> 948631600 used; max is 1190723584 >>> =A0INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving >>> 956424384 used; max is 1190723584 >>> =A0INFO 08:33:14,864 Creating new commitlog segment >>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log >>> =A0INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving >>> 972976288 used; max is 1190723584 >>> =A0INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving >>> 981829584 used; max is 1190723584 >>> =A0INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving >>> 995907192 used; max is 1190723584 >>> =A0INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving >>> 1011914464 used; max is 1190723584 >>> =A0INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408 >>> reclaimed leaving 1015729984 used; max is 1190723584 >>> =A0INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912 >>> reclaimed leaving 1019176200 used; max is 1190723584 >>> =A0INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088 >>> reclaimed leaving 1022673936 used; max is 1190723584 >>> =A0INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904 >>> reclaimed leaving 1025979696 used; max is 1190723584 >>> =A0INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072 >>> reclaimed leaving 1028992352 used; max is 1190723584 >>> =A0INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440 >>> reclaimed leaving 1031961224 used; max is 1190723584 >>> =A0INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568 >>> reclaimed leaving 1034536576 used; max is 1190723584 >>> =A0INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880 >>> reclaimed leaving 1037266456 used; max is 1190723584 >>> =A0INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432 >>> reclaimed leaving 1039709528 used; max is 1190723584 >>> =A0INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800 >>> reclaimed leaving 1042230248 used; max is 1190723584 >>> =A0INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936 >>> reclaimed leaving 1044527600 used; max is 1190723584 >>> =A0INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008 >>> reclaimed leaving 1046621520 used; max is 1190723584 >>> =A0INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336 >>> reclaimed leaving 1048677040 used; max is 1190723584 >>> =A0INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368 >>> reclaimed leaving 1050563688 used; max is 1190723584 >>> =A0INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112 >>> reclaimed leaving 1052423824 used; max is 1190723584 >>> =A0INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048 >>> reclaimed leaving 1054138992 used; max is 1190723584 >>> =A0INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560 >>> reclaimed leaving 1055761280 used; max is 1190723584 >>> =A0INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728 >>> reclaimed leaving 1057201248 used; max is 1190723584 >>> =A0INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792 >>> reclaimed leaving 1058496000 used; max is 1190723584 >>> =A0INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136 >>> reclaimed leaving 1060011424 used; max is 1190723584 >>> =A0INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792 >>> reclaimed leaving 1061388688 used; max is 1190723584 >>> =A0INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096 >>> reclaimed leaving 1062658344 used; max is 1190723584 >>> =A0INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488 >>> reclaimed leaving 1063834408 used; max is 1190723584 >>> =A0INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232 >>> reclaimed leaving 1064800992 used; max is 1190723584 >>> =A0INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288 >>> reclaimed leaving 1065798256 used; max is 1190723584 >>> =A0INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680 >>> reclaimed leaving 1066867192 used; max is 1190723584 >>> =A0INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304 >>> reclaimed leaving 1067925416 used; max is 1190723584 >>> =A0INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632 >>> reclaimed leaving 1068886912 used; max is 1190723584 >>> =A0INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184 >>> reclaimed leaving 1069705264 used; max is 1190723584 >>> =A0INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544 >>> reclaimed leaving 1070568224 used; max is 1190723584 >>> =A0INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752 >>> reclaimed leaving 1071366040 used; max is 1190723584 >>> =A0INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104 >>> reclaimed leaving 1072168400 used; max is 1190723584 >>> =A0INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264 >>> reclaimed leaving 1072743152 used; max is 1190723584 >>> =A0INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160 >>> reclaimed leaving 1073334720 used; max is 1190723584 >>> =A0INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152 >>> reclaimed leaving 1073930576 used; max is 1190723584 >>> =A0INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416 >>> reclaimed leaving 1074548160 used; max is 1190723584 >>> =A0INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544 >>> reclaimed leaving 1075122368 used; max is 1190723584 >>> =A0INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272 >>> reclaimed leaving 1075699904 used; max is 1190723584 >>> =A0INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552 >>> reclaimed leaving 1076227648 used; max is 1190723584 >>> =A0INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976 >>> reclaimed leaving 1076747648 used; max is 1190723584 >>> =A0INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584 >>> reclaimed leaving 1077152296 used; max is 1190723584 >>> =A0INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168 >>> reclaimed leaving 1077567128 used; max is 1190723584 >>> =A0INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704 >>> reclaimed leaving 1077987824 used; max is 1190723584 >>> =A0INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848 >>> reclaimed leaving 1078393448 used; max is 1190723584 >>> =A0INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136 >>> reclaimed leaving 1078708784 used; max is 1190723584 >>> =A0INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344 >>> reclaimed leaving 1079074464 used; max is 1190723584 >>> =A0INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992 >>> reclaimed leaving 1079409016 used; max is 1190723584 >>> =A0INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952 >>> reclaimed leaving 1079749672 used; max is 1190723584 >>> =A0INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768 >>> reclaimed leaving 1080014112 used; max is 1190723584 >>> =A0INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216 >>> reclaimed leaving 1080278992 used; max is 1190723584 >>> =A0INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912 >>> reclaimed leaving 1080574456 used; max is 1190723584 >>> =A0INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376 >>> reclaimed leaving 1080884096 used; max is 1190723584 >>> =A0INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512 >>> reclaimed leaving 1081076888 used; max is 1190723584 >>> =A0INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168 >>> reclaimed leaving 1081313872 used; max is 1190723584 >>> =A0INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168 >>> reclaimed leaving 1081528880 used; max is 1190723584 >>> =A0INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152 >>> reclaimed leaving 1081704648 used; max is 1190723584 >>> =A0INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216 >>> reclaimed leaving 1081889512 used; max is 1190723584 >>> =A0INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392 >>> reclaimed leaving 1082072568 used; max is 1190723584 >>> =A0INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376 >>> reclaimed leaving 1082270680 used; max is 1190723584 >>> =A0INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920 >>> reclaimed leaving 1082522872 used; max is 1190723584 >>> =A0INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392 >>> reclaimed leaving 1082632368 used; max is 1190723584 >>> =A0INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560 >>> reclaimed leaving 1082794472 used; max is 1190723584 >>> =A0INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024 >>> reclaimed leaving 1082963008 used; max is 1190723584 >>> =A0INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504 >>> reclaimed leaving 1083117048 used; max is 1190723584 >>> =A0INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192 >>> reclaimed leaving 1083213304 used; max is 1190723584 >>> =A0INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392 >>> reclaimed leaving 1083332280 used; max is 1190723584 >>> =A0INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808 >>> reclaimed leaving 1083448744 used; max is 1190723584 >>> =A0INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864 >>> reclaimed leaving 1083601864 used; max is 1190723584 >>> =A0INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520 >>> reclaimed leaving 1083680168 used; max is 1190723584 >>> =A0INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872 >>> reclaimed leaving 1083814848 used; max is 1190723584 >>> =A0INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648 >>> reclaimed leaving 1083871152 used; max is 1190723584 >>> =A0INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720 >>> reclaimed leaving 1083984728 used; max is 1190723584 >>> =A0INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056 >>> reclaimed leaving 1084063504 used; max is 1190723584 >>> >> >