Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 20039 invoked from network); 21 Apr 2010 22:06:00 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 21 Apr 2010 22:06:00 -0000 Received: (qmail 86479 invoked by uid 500); 21 Apr 2010 22:05:59 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 86449 invoked by uid 500); 21 Apr 2010 22:05:59 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 86441 invoked by uid 99); 21 Apr 2010 22:05:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:05:59 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sonnyheer@gmail.com designates 209.85.223.185 as permitted sender) Received: from [209.85.223.185] (HELO mail-iw0-f185.google.com) (209.85.223.185) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:05:53 +0000 Received: by iwn15 with SMTP id 15so5368770iwn.7 for ; Wed, 21 Apr 2010 15:05:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type :content-transfer-encoding; bh=NBPXq/u5MnP3dPXBBYr+1WLfTYxXr3E0ZKGNHwniyCg=; b=G5hZsNjxKQOlhjNuYpmVXYEoAhvVlW9n4anfIEp1S0i7BQ6eEb9hBd9/dHB7F7x8rE nQEziutT9GBEp215ShgtqSt6EYkcAbleYig0rZNFDFVDuTpmjVU89vQcQfqgU5V/bgFo +W17HV9YHuECIiw+p3eGhyyX4025bUqb463EE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=x0Y9zlv1fnzAwJGMllPBtq4Tgxh92323eRCDTtqnd4V7vlfoIgC9lT1l+AuTgHZyqz EU6hf42Mu26usHwlfxa8Ff7/H0ef9q/2f7gkiPSGw8Q6YOwgHhhL+yv9y+9Vjkh+7tcu +T0q1AtUwkhrGhyAIlDqNICgdUNI4mrSRYF2E= MIME-Version: 1.0 Received: by 10.231.15.133 with HTTP; Wed, 21 Apr 2010 15:05:32 -0700 (PDT) In-Reply-To: References: Date: Wed, 21 Apr 2010 15:05:32 -0700 Received: by 10.231.157.208 with SMTP id c16mr3038481ibx.81.1271887532397; Wed, 21 Apr 2010 15:05:32 -0700 (PDT) Message-ID: Subject: Re: Import using cassandra 0.6.1 From: Sonny Heer To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable What does OOM stand for? for a given insert the size is small (meaning the a single insert operation only has about a sentence of data) although as the insert process continues, the columns under a given row key could potentially grow to be large. Is that what you mean? An operation entails: Read Insert row (IE: rowkey: FOO columnName: BLAH value: 10) Delete a row Insert different row (IE: rowkey: FOO columnName: 0000010|Blah value: 10) millions of these operations are performed in sequence as files are read from a directory source. On Wed, Apr 21, 2010 at 2:37 PM, Jonathan Ellis wrote: > then that's not the problem. > > are you writing large rows that OOM during compaction? > > On Wed, Apr 21, 2010 at 4:34 PM, Sonny Heer wrote: >> They are showing up as completed? =A0Is this correct: >> >> >> Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending =A0 = =A0 =A0Completed >> STREAM-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> RESPONSE-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 = 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> ROW-READ-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 = 0 =A0 =A0 =A0 =A0 517446 >> LB-OPERATIONS =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 = 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> MESSAGE-DESERIALIZER-POOL =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A00 >> GMFD =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 = =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> LB-TARGET =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> CONSISTENCY-MANAGER =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 >> ROW-MUTATION-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A01353622 >> MESSAGE-STREAMING-POOL =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 >> LOAD-BALANCER-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 >> FLUSH-SORTER-POOL =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> MEMTABLE-POST-FLUSHER =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 >> FLUSH-WRITER-POOL =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> AE-SERVICE-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A0 =A0 =A0 =A00 >> >> >> On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis wrot= e: >>> you need to figure out where the memory is going. =A0check tpstats, if >>> the pending ops are large somewhere that means you're just generating >>> insert ops faster than it can handle. >>> >>> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer wrote= : >>>> note: I'm using the Thrift API to insert. =A0The commitLog directory >>>> continues to grow. =A0The heap size continues to grow as well. >>>> >>>> I decreased MemtableSizeInMB size, but noticed no changes. =A0Any idea >>>> what is causing this, and/or what property i need to tweek to >>>> alleviate this? =A0What is the "insert threshold"? >>>> >>>> I moved to a more powerful node as well, it still ended up failing >>>> just after a longer period. >>>> >>>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis w= rote: >>>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts >>>>> >>>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer wr= ote: >>>>>> Currently running on a single node with intensive write operations. >>>>>> >>>>>> >>>>>> After running for a while... >>>>>> >>>>>> Client starts outputting: >>>>>> >>>>>> TimedOutException() >>>>>> =A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$insert_resul= t.read(Cassandra.java:12232) >>>>>> =A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$Client.recv_= insert(Cassandra.java:670) >>>>>> =A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$Client.inser= t(Cassandra.java:643) >>>>>> >>>>>> Cassandra starts outputting: >>>>>> >>>>>> =A0INFO 08:08:49,864 Cassandra starting up... >>>>>> =A0INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leavi= ng >>>>>> 220554976 used; max is 1190723584 >>>>>> =A0INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leavi= ng >>>>>> 230245792 used; max is 1190723584 >>>>>> =A0INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leavi= ng >>>>>> 250127792 used; max is 1190723584 >>>>>> =A0INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leavi= ng >>>>>> 261657720 used; max is 1190723584 >>>>>> =A0INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leavi= ng >>>>>> 273764040 used; max is 1190723584 >>>>>> =A0INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leavi= ng >>>>>> 282912656 used; max is 1190723584 >>>>>> =A0INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leavi= ng >>>>>> 294861824 used; max is 1190723584 >>>>>> =A0INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leavi= ng >>>>>> 304045640 used; max is 1190723584 >>>>>> =A0INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leavi= ng >>>>>> 312964344 used; max is 1190723584 >>>>>> =A0INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leavi= ng >>>>>> 324961320 used; max is 1190723584 >>>>>> =A0INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leavi= ng >>>>>> 345874144 used; max is 1190723584 >>>>>> =A0INFO 08:20:27,357 Creating new commitlog segment >>>>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log >>>>>> =A0INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leavi= ng >>>>>> 366587984 used; max is 1190723584 >>>>>> =A0INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leavi= ng >>>>>> 384059832 used; max is 1190723584 >>>>>> =A0INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leavi= ng >>>>>> 383631432 used; max is 1190723584 >>>>>> =A0INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leavi= ng >>>>>> 399697888 used; max is 1190723584 >>>>>> =A0INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leavi= ng >>>>>> 415428952 used; max is 1190723584 >>>>>> =A0INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leavi= ng >>>>>> 433542656 used; max is 1190723584 >>>>>> =A0INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leavi= ng >>>>>> 452030560 used; max is 1190723584 >>>>>> =A0INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leavi= ng >>>>>> 464013992 used; max is 1190723584 >>>>>> =A0INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leavi= ng >>>>>> 471978840 used; max is 1190723584 >>>>>> =A0INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leavi= ng >>>>>> 475912120 used; max is 1190723584 >>>>>> =A0INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leavi= ng >>>>>> 483742344 used; max is 1190723584 >>>>>> =A0INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leavi= ng >>>>>> 491761672 used; max is 1190723584 >>>>>> =A0INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leavi= ng >>>>>> 507442800 used; max is 1190723584 >>>>>> =A0INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leavi= ng >>>>>> 523028304 used; max is 1190723584 >>>>>> =A0INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leavi= ng >>>>>> 530865568 used; max is 1190723584 >>>>>> =A0INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leavi= ng >>>>>> 542690216 used; max is 1190723584 >>>>>> =A0INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leavi= ng >>>>>> 550728232 used; max is 1190723584 >>>>>> =A0INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leavi= ng >>>>>> 573981584 used; max is 1190723584 >>>>>> =A0INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leavi= ng >>>>>> 563891472 used; max is 1190723584 >>>>>> =A0INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leavi= ng >>>>>> 575756040 used; max is 1190723584 >>>>>> =A0INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leavi= ng >>>>>> 583631432 used; max is 1190723584 >>>>>> =A0INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leavi= ng >>>>>> 595405816 used; max is 1190723584 >>>>>> =A0INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leavi= ng >>>>>> 603389432 used; max is 1190723584 >>>>>> =A0INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leavi= ng >>>>>> 615150584 used; max is 1190723584 >>>>>> =A0INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leavi= ng >>>>>> 623125104 used; max is 1190723584 >>>>>> =A0INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leavi= ng >>>>>> 634661008 used; max is 1190723584 >>>>>> =A0INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leavi= ng >>>>>> 646272840 used; max is 1190723584 >>>>>> =A0INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leavi= ng >>>>>> 653978688 used; max is 1190723584 >>>>>> =A0INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leavi= ng >>>>>> 665462760 used; max is 1190723584 >>>>>> =A0INFO 08:26:44,895 Creating new commitlog segment >>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log >>>>>> =A0INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leavi= ng >>>>>> 673423168 used; max is 1190723584 >>>>>> =A0INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leavi= ng >>>>>> 685277616 used; max is 1190723584 >>>>>> =A0INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leavi= ng >>>>>> 692803112 used; max is 1190723584 >>>>>> =A0INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leavi= ng >>>>>> 700213488 used; max is 1190723584 >>>>>> =A0INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leavi= ng >>>>>> 708002984 used; max is 1190723584 >>>>>> =A0INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leavi= ng >>>>>> 719547672 used; max is 1190723584 >>>>>> =A0INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leavi= ng >>>>>> 727261696 used; max is 1190723584 >>>>>> =A0INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leavi= ng >>>>>> 738760480 used; max is 1190723584 >>>>>> =A0INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leavi= ng >>>>>> 746376392 used; max is 1190723584 >>>>>> =A0INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leavi= ng >>>>>> 754102376 used; max is 1190723584 >>>>>> =A0INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leavi= ng >>>>>> 757926896 used; max is 1190723584 >>>>>> =A0INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leavi= ng >>>>>> 763964048 used; max is 1190723584 >>>>>> =A0INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leavi= ng >>>>>> 771496792 used; max is 1190723584 >>>>>> =A0INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leavi= ng >>>>>> 779088496 used; max is 1190723584 >>>>>> =A0INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leavi= ng >>>>>> 786858536 used; max is 1190723584 >>>>>> =A0INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leavi= ng >>>>>> 794395528 used; max is 1190723584 >>>>>> =A0INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leavi= ng >>>>>> 805862000 used; max is 1190723584 >>>>>> =A0INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leavi= ng >>>>>> 813708392 used; max is 1190723584 >>>>>> =A0INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leavi= ng >>>>>> 821245568 used; max is 1190723584 >>>>>> =A0INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leavi= ng >>>>>> 828614768 used; max is 1190723584 >>>>>> =A0INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leavi= ng >>>>>> 835962544 used; max is 1190723584 >>>>>> =A0INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leavi= ng >>>>>> 844183384 used; max is 1190723584 >>>>>> =A0INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leavi= ng >>>>>> 847924056 used; max is 1190723584 >>>>>> =A0INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leavi= ng >>>>>> 841384040 used; max is 1190723584 >>>>>> =A0INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leavi= ng >>>>>> 839554448 used; max is 1190723584 >>>>>> =A0INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leavi= ng >>>>>> 850851600 used; max is 1190723584 >>>>>> =A0INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leavi= ng >>>>>> 869886032 used; max is 1190723584 >>>>>> =A0INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leavi= ng >>>>>> 881346312 used; max is 1190723584 >>>>>> =A0INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leavi= ng >>>>>> 892677840 used; max is 1190723584 >>>>>> =A0INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leavi= ng >>>>>> 911084096 used; max is 1190723584 >>>>>> =A0INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leavi= ng >>>>>> 922370800 used; max is 1190723584 >>>>>> =A0INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leavi= ng >>>>>> 929916416 used; max is 1190723584 >>>>>> =A0INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leavi= ng >>>>>> 941201152 used; max is 1190723584 >>>>>> =A0INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leavi= ng >>>>>> 948631600 used; max is 1190723584 >>>>>> =A0INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leavi= ng >>>>>> 956424384 used; max is 1190723584 >>>>>> =A0INFO 08:33:14,864 Creating new commitlog segment >>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log >>>>>> =A0INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leavi= ng >>>>>> 972976288 used; max is 1190723584 >>>>>> =A0INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leavi= ng >>>>>> 981829584 used; max is 1190723584 >>>>>> =A0INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leavi= ng >>>>>> 995907192 used; max is 1190723584 >>>>>> =A0INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leavi= ng >>>>>> 1011914464 used; max is 1190723584 >>>>>> =A0INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408 >>>>>> reclaimed leaving 1015729984 used; max is 1190723584 >>>>>> =A0INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912 >>>>>> reclaimed leaving 1019176200 used; max is 1190723584 >>>>>> =A0INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088 >>>>>> reclaimed leaving 1022673936 used; max is 1190723584 >>>>>> =A0INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904 >>>>>> reclaimed leaving 1025979696 used; max is 1190723584 >>>>>> =A0INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072 >>>>>> reclaimed leaving 1028992352 used; max is 1190723584 >>>>>> =A0INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440 >>>>>> reclaimed leaving 1031961224 used; max is 1190723584 >>>>>> =A0INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568 >>>>>> reclaimed leaving 1034536576 used; max is 1190723584 >>>>>> =A0INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880 >>>>>> reclaimed leaving 1037266456 used; max is 1190723584 >>>>>> =A0INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432 >>>>>> reclaimed leaving 1039709528 used; max is 1190723584 >>>>>> =A0INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800 >>>>>> reclaimed leaving 1042230248 used; max is 1190723584 >>>>>> =A0INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936 >>>>>> reclaimed leaving 1044527600 used; max is 1190723584 >>>>>> =A0INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008 >>>>>> reclaimed leaving 1046621520 used; max is 1190723584 >>>>>> =A0INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336 >>>>>> reclaimed leaving 1048677040 used; max is 1190723584 >>>>>> =A0INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368 >>>>>> reclaimed leaving 1050563688 used; max is 1190723584 >>>>>> =A0INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112 >>>>>> reclaimed leaving 1052423824 used; max is 1190723584 >>>>>> =A0INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048 >>>>>> reclaimed leaving 1054138992 used; max is 1190723584 >>>>>> =A0INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560 >>>>>> reclaimed leaving 1055761280 used; max is 1190723584 >>>>>> =A0INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728 >>>>>> reclaimed leaving 1057201248 used; max is 1190723584 >>>>>> =A0INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792 >>>>>> reclaimed leaving 1058496000 used; max is 1190723584 >>>>>> =A0INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136 >>>>>> reclaimed leaving 1060011424 used; max is 1190723584 >>>>>> =A0INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792 >>>>>> reclaimed leaving 1061388688 used; max is 1190723584 >>>>>> =A0INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096 >>>>>> reclaimed leaving 1062658344 used; max is 1190723584 >>>>>> =A0INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488 >>>>>> reclaimed leaving 1063834408 used; max is 1190723584 >>>>>> =A0INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232 >>>>>> reclaimed leaving 1064800992 used; max is 1190723584 >>>>>> =A0INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288 >>>>>> reclaimed leaving 1065798256 used; max is 1190723584 >>>>>> =A0INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680 >>>>>> reclaimed leaving 1066867192 used; max is 1190723584 >>>>>> =A0INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304 >>>>>> reclaimed leaving 1067925416 used; max is 1190723584 >>>>>> =A0INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632 >>>>>> reclaimed leaving 1068886912 used; max is 1190723584 >>>>>> =A0INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184 >>>>>> reclaimed leaving 1069705264 used; max is 1190723584 >>>>>> =A0INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544 >>>>>> reclaimed leaving 1070568224 used; max is 1190723584 >>>>>> =A0INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752 >>>>>> reclaimed leaving 1071366040 used; max is 1190723584 >>>>>> =A0INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104 >>>>>> reclaimed leaving 1072168400 used; max is 1190723584 >>>>>> =A0INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264 >>>>>> reclaimed leaving 1072743152 used; max is 1190723584 >>>>>> =A0INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160 >>>>>> reclaimed leaving 1073334720 used; max is 1190723584 >>>>>> =A0INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152 >>>>>> reclaimed leaving 1073930576 used; max is 1190723584 >>>>>> =A0INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416 >>>>>> reclaimed leaving 1074548160 used; max is 1190723584 >>>>>> =A0INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544 >>>>>> reclaimed leaving 1075122368 used; max is 1190723584 >>>>>> =A0INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272 >>>>>> reclaimed leaving 1075699904 used; max is 1190723584 >>>>>> =A0INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552 >>>>>> reclaimed leaving 1076227648 used; max is 1190723584 >>>>>> =A0INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976 >>>>>> reclaimed leaving 1076747648 used; max is 1190723584 >>>>>> =A0INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584 >>>>>> reclaimed leaving 1077152296 used; max is 1190723584 >>>>>> =A0INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168 >>>>>> reclaimed leaving 1077567128 used; max is 1190723584 >>>>>> =A0INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704 >>>>>> reclaimed leaving 1077987824 used; max is 1190723584 >>>>>> =A0INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848 >>>>>> reclaimed leaving 1078393448 used; max is 1190723584 >>>>>> =A0INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136 >>>>>> reclaimed leaving 1078708784 used; max is 1190723584 >>>>>> =A0INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344 >>>>>> reclaimed leaving 1079074464 used; max is 1190723584 >>>>>> =A0INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992 >>>>>> reclaimed leaving 1079409016 used; max is 1190723584 >>>>>> =A0INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952 >>>>>> reclaimed leaving 1079749672 used; max is 1190723584 >>>>>> =A0INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768 >>>>>> reclaimed leaving 1080014112 used; max is 1190723584 >>>>>> =A0INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216 >>>>>> reclaimed leaving 1080278992 used; max is 1190723584 >>>>>> =A0INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912 >>>>>> reclaimed leaving 1080574456 used; max is 1190723584 >>>>>> =A0INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376 >>>>>> reclaimed leaving 1080884096 used; max is 1190723584 >>>>>> =A0INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512 >>>>>> reclaimed leaving 1081076888 used; max is 1190723584 >>>>>> =A0INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168 >>>>>> reclaimed leaving 1081313872 used; max is 1190723584 >>>>>> =A0INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168 >>>>>> reclaimed leaving 1081528880 used; max is 1190723584 >>>>>> =A0INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152 >>>>>> reclaimed leaving 1081704648 used; max is 1190723584 >>>>>> =A0INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216 >>>>>> reclaimed leaving 1081889512 used; max is 1190723584 >>>>>> =A0INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392 >>>>>> reclaimed leaving 1082072568 used; max is 1190723584 >>>>>> =A0INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376 >>>>>> reclaimed leaving 1082270680 used; max is 1190723584 >>>>>> =A0INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920 >>>>>> reclaimed leaving 1082522872 used; max is 1190723584 >>>>>> =A0INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392 >>>>>> reclaimed leaving 1082632368 used; max is 1190723584 >>>>>> =A0INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560 >>>>>> reclaimed leaving 1082794472 used; max is 1190723584 >>>>>> =A0INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024 >>>>>> reclaimed leaving 1082963008 used; max is 1190723584 >>>>>> =A0INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504 >>>>>> reclaimed leaving 1083117048 used; max is 1190723584 >>>>>> =A0INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192 >>>>>> reclaimed leaving 1083213304 used; max is 1190723584 >>>>>> =A0INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392 >>>>>> reclaimed leaving 1083332280 used; max is 1190723584 >>>>>> =A0INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808 >>>>>> reclaimed leaving 1083448744 used; max is 1190723584 >>>>>> =A0INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864 >>>>>> reclaimed leaving 1083601864 used; max is 1190723584 >>>>>> =A0INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520 >>>>>> reclaimed leaving 1083680168 used; max is 1190723584 >>>>>> =A0INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872 >>>>>> reclaimed leaving 1083814848 used; max is 1190723584 >>>>>> =A0INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648 >>>>>> reclaimed leaving 1083871152 used; max is 1190723584 >>>>>> =A0INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720 >>>>>> reclaimed leaving 1083984728 used; max is 1190723584 >>>>>> =A0INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056 >>>>>> reclaimed leaving 1084063504 used; max is 1190723584 >>>>>> >>>>> >>>> >>> >> >