Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 67948 invoked from network); 29 Mar 2010 02:09:11 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 29 Mar 2010 02:09:11 -0000 Received: (qmail 94873 invoked by uid 500); 29 Mar 2010 02:09:09 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 94828 invoked by uid 500); 29 Mar 2010 02:09:08 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 94820 invoked by uid 99); 29 Mar 2010 02:09:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 02:09:08 +0000 X-ASF-Spam-Status: No, hits=1.7 required=10.0 tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of oss.akk@gmail.com designates 209.85.160.48 as permitted sender) Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 02:08:58 +0000 Received: by pwi7 with SMTP id 7so7423352pwi.35 for ; Sun, 28 Mar 2010 19:08:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type; bh=Q1BTQY5HmdUQS0RWrRyT5l82qhWJf+FdmvhhYXN6fxQ=; b=PLmOIgrwh+tUJqpHfxb1s0LNmcXaAObfqGHrfw1PF73gEQuC5QFm5pJ6W3TN2P9Bbv K4UdbrjyirwLhDGMcpsBBPJvAroPfSgIO3WdJbyktgKolIwgURB02vOInqRy4anzN9H1 NW+O/X1Ntbpq6aqTY3eMdINL3Ml33Ah+D2Qho= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=ppuCk9PHfEOLWSKxFQ2QEbevcFxRjvcZqmapA3vSrxHUynJO23r2/HwYVE/2Eohf3N CvzkBMItKNy6tmz9Tr60UZ6pcdhyF2avi4bfl6c/VmFyIzvNRgl2YbrAtyG4tl3m2lFE QWuLiAsOaJo1KHErqXMbVaYjY7w5B2XRDlaj4= MIME-Version: 1.0 Received: by 10.142.116.19 with HTTP; Sun, 28 Mar 2010 19:08:38 -0700 (PDT) In-Reply-To: <4BAD1D05.9070201@jobdiva.com> References: <4BAA563A.7050302@jobdiva.com> <4BACC5B1.606@jobdiva.com> <4BACCCBE.7050502@jobdiva.com> <9ac0c6aa1003261139p6babb2c7td5f0774bbfcfebf4@mail.gmail.com> <4BAD0C9A.1030208@jobdiva.com> <9ac0c6aa1003261253o7d431c18o4edc13dd61dce62f@mail.gmail.com> <4BAD13B6.7060400@jobdiva.com> <9ac0c6aa1003261334t57cd12b3qab7eb27b64063260@mail.gmail.com> <4BAD1C2D.6090705@jobdiva.com> <4BAD1D05.9070201@jobdiva.com> Date: Sun, 28 Mar 2010 19:08:38 -0700 Received: by 10.142.60.18 with SMTP id i18mr1695406wfa.275.1269828518239; Sun, 28 Mar 2010 19:08:38 -0700 (PDT) Message-ID: Subject: Re: Garbage Collection performance on 2.9.2 From: Karthik K To: java-user@lucene.apache.org Cc: Michael McCandless Content-Type: multipart/alternative; boundary=00504502ad20c162160482e6fd2d --00504502ad20c162160482e6fd2d Content-Type: text/plain; charset=ISO-8859-1 Which version of jdk are you using ? We are seeing some problems (with GC) with jdk6 u18 , and reverted back to u14 (similar problems reported by hadoop / hbase community as well). On Fri, Mar 26, 2010 at 1:45 PM, Siraj Haider wrote: > I am sorry for replying many time, but I just thought it might be a good > idea to show you the code I have to open an index. I know it might not be a > very clean and efficient piece of code, so I am open to suggestions on how > to improve it. > > / public void openIndex() throws Exception { > try { > if (which_is==0) { > // first time > > // unlock the index, if its locked > if (IndexReader.indexExists(fsDir) && (IndexWriter.isLocked(fsDir))) > IndexWriter.unlock(fsDir); > is_1 = new IndexSearcher(fsDir, true); > > is_1.setSimilarity(new JobdivaSimilarity()); > is_1 = warmUp(is_1); > which_is=1; > > } else if (which_is==1) { > try { > // is_1 is the current is, now need to change to is_2 > if (is_2!=null) is_2.close(); > is_2=null; > } catch (Exception e) { } > //System.gc(); > > // unlock the index, if its locked > if (IndexReader.indexExists(fsDir) && (IndexWriter.isLocked(fsDir))) > IndexWriter.unlock(fsDir); > is_2 = new IndexSearcher(fsDir, true); > is_2.setSimilarity(new JobdivaSimilarity()); > is_2 = warmUp(is_2); > which_is=2; > > //give some time to LuceneResults (searches already ran against > is_1) to finish before closing IndexSearcher > Thread.sleep(_indexReOpeningWait*1000); > try { > if (is_1!=null) is_1.close(); > is_1=null; > } catch (Exception e) { } > > } else if (which_is==2) { > try { > // is_2 is the current is, now need to change to is_1 > if (is_1!=null) is_1.close(); > is_1=null; > } catch (Exception e) { } > //System.gc(); > > // unlock the index, if its locked > if (IndexReader.indexExists(fsDir) && (IndexWriter.isLocked(fsDir))) > IndexWriter.unlock(fsDir); > is_1 = new IndexSearcher(fsDir, true); > is_1.setSimilarity(new JobdivaSimilarity()); > is_1 = warmUp(is_1); > which_is=1; > > //give some time to LuceneResults (searches already ran against > is_2) to finish before closing IndexSearcher > Thread.sleep(_indexReOpeningWait*1000); > try { > if (is_2!=null) is_2.close(); > is_2=null; > } catch (Exception e) { } > > } > //System.gc(); > print("INDEX OPENED"); > } catch (Exception e) { > e.printStackTrace(System.out); > } > }/ > > > On 3/26/2010 4:42 PM, Siraj Haider wrote: > >> I have one question regardning reopening the index. Currently I have >> two index searchers that I switch back and forth whenever I need to open >> an index again. Keep in mind that my searchers are opened from >> FSDirectory. Now if I want to introduce IndexReader in the mix, do I >> need to keep two IndexesReaders for each of my IndexSearcher? And if I >> am keeping two readers, will it make a difference in garbage collection? >> because now I will have two readers open all the time. >> >> -siraj >> >> On 3/26/2010 4:34 PM, Michael McCandless wrote: >> >> >>> OK but please first switch to IndexReader.reopen :) >>> >>> Mike >>> >>> On Fri, Mar 26, 2010 at 4:06 PM, Siraj Haider >>> wrote: >>> >>> >>> >>>> Mike, >>>> I just found out another problem that was recently introduced. We do a >>>> radius search based on the zipcode and we do it on the results we have >>>> received from the query. I recently tried to change it to a filter and >>>> that >>>> was causing a lot of slowness in the search and may be generating more >>>> garbage. I switched it back to the post-search filtering and now >>>> searching >>>> is back to the regular speed and the machines seem to be stable and >>>> generating similar garbage. But we still have this problem of frequent >>>> full >>>> gc cycles even on 2.3.2 indexes occasionally. I will try to get a dump >>>> from >>>> a machine if it faces this problem again. Thanks again for your help. >>>> >>>> regards >>>> -siraj >>>> >>>> >>>> On 3/26/2010 3:53 PM, Michael McCandless wrote: >>>> >>>> >>>> >>>>> OK. >>>>> >>>>> Doing a full open every time will create more garbage than necessary... >>>>> >>>>> But, that should've also been the case with 2.3 as well. >>>>> >>>>> Mike >>>>> >>>>> On Fri, Mar 26, 2010 at 3:35 PM, Siraj Haider >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Right now I dont reopen the reader. Infact, I am opening my searcher >>>>>> from >>>>>> FSDirectory. I will try to get the dump when the machine is thrashing >>>>>> and >>>>>> will provide you with that. Thanks so much for helping me on this. >>>>>> >>>>>> -siraj >>>>>> >>>>>> >>>>>> >>>>>> On 3/26/2010 2:39 PM, Michael McCandless wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Are you using IndexReader.reopen? >>>>>>> >>>>>>> Looks like your heap is dominated by ~6 million String and their >>>>>>> char[]s. >>>>>>> >>>>>>> The terms index (~2.8 million terms, so almost half of the String >>>>>>> instances) is also taking up a fair amount of RAM, but that should be >>>>>>> relatively static (assuming you're using reopen). >>>>>>> >>>>>>> Are you loading many stored fields per doc, for many hits per query? >>>>>>> That'd generate garbage, but shouldn't be moreso than in 2.3.x. >>>>>>> >>>>>>> If you can get a heap dump on the "thrashing" machine maybe >>>>>>> that'll prove more telling ... but nothing obvious stands out in >>>>>>> this one. >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>> On Fri, Mar 26, 2010 at 11:03 AM, Siraj Haider >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Here is some more information to what my settings are: >>>>>>>> >>>>>>>> *JVM Options* >>>>>>>> /-server -verbose:gc -XX:+PrintClassHistogram >>>>>>>> -XX:MaxGCPauseMillis=1500 >>>>>>>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC >>>>>>>> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails >>>>>>>> -XX:+PrintGCTimeStamps >>>>>>>> -Xms2048m -Xmx2048m -Dsun.rmi.dgc.client.gcInterval=3600000 >>>>>>>> -Dsun.rmi.dgc.server.gcInterval=3600000 / >>>>>>>> >>>>>>>> *And I see the following gc output very frequently in my tomcat >>>>>>>> output:* >>>>>>>> >>>>>>>> 1841.600: [GC 1841.600: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 616576 bytes, 616576 total >>>>>>>> - age 2: 595192 bytes, 1211768 total >>>>>>>> - age 3: 4029632 bytes, 5241400 total >>>>>>>> - age 4: 213960 bytes, 5455360 total >>>>>>>> : 109370K->5554K(118016K), 0.0077710 secs] >>>>>>>> 1518434K->1414720K(2084096K), >>>>>>>> 0.0078990 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1842.067: [GC 1842.067: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 118968 bytes, 118968 total >>>>>>>> - age 2: 1091472 bytes, 1210440 total >>>>>>>> - age 3: 595192 bytes, 1805632 total >>>>>>>> - age 4: 4029632 bytes, 5835264 total >>>>>>>> : 110514K->5946K(118016K), 0.0084010 secs] >>>>>>>> 1519680K->1415321K(2084096K), >>>>>>>> 0.0085220 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1842.535: [GC 1842.536: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 513960 bytes, 513960 total >>>>>>>> - age 2: 113608 bytes, 627568 total >>>>>>>> - age 3: 613008 bytes, 1240576 total >>>>>>>> - age 4: 595192 bytes, 1835768 total >>>>>>>> : 110906K->2102K(118016K), 0.0080120 secs] >>>>>>>> 1520281K->1414435K(2084096K), >>>>>>>> 0.0081330 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1842.812: [CMS-concurrent-mark: 7.065/7.811 secs] [Times: user=24.12 >>>>>>>> sys=4.06, real=7.82 secs] >>>>>>>> 1842.812: [CMS-concurrent-preclean-start] >>>>>>>> 1843.004: [GC 1843.004: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 16280 bytes, 16280 total >>>>>>>> - age 2: 990336 bytes, 1006616 total >>>>>>>> - age 3: 113608 bytes, 1120224 total >>>>>>>> - age 4: 613008 bytes, 1733232 total >>>>>>>> : 107062K->1909K(118016K), 0.0065890 secs] >>>>>>>> 1519395K->1414823K(2084096K), >>>>>>>> 0.0067090 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1843.392: [CMS-concurrent-preclean: 0.509/0.580 secs] [Times: >>>>>>>> user=1.09 >>>>>>>> sys=0.15, real=0.58 secs] >>>>>>>> 1843.392: [CMS-concurrent-abortable-preclean-start] >>>>>>>> 1843.455: [GC 1843.455: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 149928 bytes, 149928 total >>>>>>>> - age 2: 9440 bytes, 159368 total >>>>>>>> - age 3: 990336 bytes, 1149704 total >>>>>>>> - age 4: 113608 bytes, 1263312 total >>>>>>>> : 106869K->1323K(118016K), 0.0066280 secs] >>>>>>>> 1519783K->1414836K(2084096K), >>>>>>>> 0.0067490 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1843.461: [CMS-concurrent-abortable-preclean: 0.006/0.069 secs] >>>>>>>> [Times: >>>>>>>> user=0.11 sys=0.01, real=0.07 secs] >>>>>>>> 1843.462: [GC[YG occupancy: 3354 K (118016 K)]1843.462: [Rescan >>>>>>>> (parallel) , >>>>>>>> 0.0023630 secs]1843.464: [weak refs processing, 0.0005650 secs] [1 >>>>>>>> CMS-remark: 1413512K(1966080K)] 1416867K(2084096K), 0.0031210 secs] >>>>>>>> [Times: >>>>>>>> user=0.02 sys=0.00, real=0.00 secs] >>>>>>>> 1843.465: [CMS-concurrent-sweep-start] >>>>>>>> 1843.916: [GC 1843.916: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2107208 bytes, 2107208 total >>>>>>>> - age 2: 118624 bytes, 2225832 total >>>>>>>> - age 3: 9440 bytes, 2235272 total >>>>>>>> - age 4: 511920 bytes, 2747192 total >>>>>>>> : 106283K->2844K(118016K), 0.0070550 secs] >>>>>>>> 1472611K->1369283K(2084096K), >>>>>>>> 0.0071890 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1844.380: [GC 1844.380: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 45232 bytes, 45232 total >>>>>>>> - age 2: 2094312 bytes, 2139544 total >>>>>>>> - age 3: 118624 bytes, 2258168 total >>>>>>>> - age 4: 9440 bytes, 2267608 total >>>>>>>> : 107804K->2308K(118016K), 0.0090090 secs] >>>>>>>> 1417654K->1312657K(2084096K), >>>>>>>> 0.0091320 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] >>>>>>>> 1844.829: [GC 1844.829: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 104808 bytes, 104808 total >>>>>>>> - age 2: 41984 bytes, 146792 total >>>>>>>> - age 3: 2094152 bytes, 2240944 total >>>>>>>> - age 4: 118624 bytes, 2359568 total >>>>>>>> : 107268K->2403K(118016K), 0.0092200 secs] >>>>>>>> 1349534K->1244678K(2084096K), >>>>>>>> 0.0093420 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] >>>>>>>> 1845.304: [GC 1845.304: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1665184 bytes, 1665184 total >>>>>>>> - age 2: 103592 bytes, 1768776 total >>>>>>>> - age 3: 41984 bytes, 1810760 total >>>>>>>> - age 4: 1593816 bytes, 3404576 total >>>>>>>> : 107363K->3481K(118016K), 0.0081660 secs] >>>>>>>> 1305631K->1201864K(2084096K), >>>>>>>> 0.0082860 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] >>>>>>>> 1845.856: [GC 1845.856: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 85520 bytes, 85520 total >>>>>>>> - age 2: 1662552 bytes, 1748072 total >>>>>>>> - age 3: 103592 bytes, 1851664 total >>>>>>>> - age 4: 41984 bytes, 1893648 total >>>>>>>> : 108441K->2020K(118016K), 0.0083700 secs] >>>>>>>> 1213393K->1108529K(2084096K), >>>>>>>> 0.0084880 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] >>>>>>>> 1846.395: [GC 1846.395: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 28824 bytes, 28824 total >>>>>>>> - age 2: 83384 bytes, 112208 total >>>>>>>> - age 3: 1662504 bytes, 1774712 total >>>>>>>> - age 4: 103496 bytes, 1878208 total >>>>>>>> : 106980K->1999K(118016K), 0.0067460 secs] >>>>>>>> 1105216K->1000276K(2084096K), >>>>>>>> 0.0068650 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] >>>>>>>> 1846.445: [CMS-concurrent-sweep: 2.921/2.979 secs] [Times: user=5.97 >>>>>>>> sys=0.37, real=2.98 secs] >>>>>>>> 1846.445: [CMS-concurrent-reset-start] >>>>>>>> 1846.486: [CMS-concurrent-reset: 0.041/0.041 secs] [Times: user=0.09 >>>>>>>> sys=0.01, real=0.04 secs] >>>>>>>> 1846.951: [GC 1846.951: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 81680 bytes, 81680 total >>>>>>>> - age 2: 15944 bytes, 97624 total >>>>>>>> - age 3: 83328 bytes, 180952 total >>>>>>>> - age 4: 1662504 bytes, 1843456 total >>>>>>>> : 106959K->1882K(118016K), 0.0074460 secs] >>>>>>>> 1103481K->998505K(2084096K), >>>>>>>> 0.0075870 secs] [Times: user=0.05 sys=0.01, real=0.00 secs] >>>>>>>> 1847.409: [GC 1847.409: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 39184 bytes, 39184 total >>>>>>>> - age 2: 80024 bytes, 119208 total >>>>>>>> - age 3: 15824 bytes, 135032 total >>>>>>>> - age 4: 83264 bytes, 218296 total >>>>>>>> : 106842K->337K(118016K), 0.0066910 secs] >>>>>>>> 1103465K->998584K(2084096K), >>>>>>>> 0.0067920 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1847.863: [GC 1847.863: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 752 bytes, 752 total >>>>>>>> - age 2: 35624 bytes, 36376 total >>>>>>>> - age 3: 80024 bytes, 116400 total >>>>>>>> - age 4: 15824 bytes, 132224 total >>>>>>>> : 105297K->198K(118016K), 0.0055200 secs] >>>>>>>> 1103544K->998527K(2084096K), >>>>>>>> 0.0056140 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1848.326: [GC 1848.326: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 599496 bytes, 599496 total >>>>>>>> - age 3: 35272 bytes, 634768 total >>>>>>>> - age 4: 80024 bytes, 714792 total >>>>>>>> : 105158K->725K(118016K), 0.0059760 secs] >>>>>>>> 1103487K->999069K(2084096K), >>>>>>>> 0.0060730 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1848.677: [GC 1848.677: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1457672 bytes, 1457672 total >>>>>>>> - age 2: 597560 bytes, 2055232 total >>>>>>>> - age 4: 35120 bytes, 2090352 total >>>>>>>> : 105685K->2097K(118016K), 0.0061090 secs] >>>>>>>> 1104029K->1000519K(2084096K), >>>>>>>> 0.0062260 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] >>>>>>>> 1848.892: [GC 1848.892: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4072 bytes, 4072 total >>>>>>>> - age 2: 1446824 bytes, 1450896 total >>>>>>>> - age 3: 597560 bytes, 2048456 total >>>>>>>> : 107057K->2093K(118016K), 0.0056390 secs] >>>>>>>> 1105479K->1000550K(2084096K), >>>>>>>> 0.0057550 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1849.102: [GC 1849.102: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4168 bytes, 4168 total >>>>>>>> - age 2: 160 bytes, 4328 total >>>>>>>> - age 3: 1446824 bytes, 1451152 total >>>>>>>> - age 4: 597560 bytes, 2048712 total >>>>>>>> : 107053K->2151K(118016K), 0.0057770 secs] >>>>>>>> 1105510K->1000607K(2084096K), >>>>>>>> 0.0058840 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1849.312: [GC 1849.312: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3512 bytes, 3512 total >>>>>>>> - age 3: 160 bytes, 3672 total >>>>>>>> - age 4: 1446824 bytes, 1450496 total >>>>>>>> : 107111K->1629K(118016K), 0.0057300 secs] >>>>>>>> 1105567K->1000669K(2084096K), >>>>>>>> 0.0058430 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1849.523: [GC 1849.523: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5432 bytes, 5432 total >>>>>>>> - age 4: 160 bytes, 5592 total >>>>>>>> : 106589K->132K(118016K), 0.0062260 secs] >>>>>>>> 1105629K->1000014K(2084096K), >>>>>>>> 0.0063410 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1849.734: [GC 1849.734: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2504 bytes, 2504 total >>>>>>>> - age 2: 864 bytes, 3368 total >>>>>>>> : 105092K->15K(118016K), 0.0072610 secs] >>>>>>>> 1104974K->999897K(2084096K), >>>>>>>> 0.0073600 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1849.958: [GC 1849.958: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1823840 bytes, 1823840 total >>>>>>>> - age 3: 864 bytes, 1824704 total >>>>>>>> : 104975K->1788K(118016K), 0.0066770 secs] >>>>>>>> 1104857K->1001670K(2084096K), >>>>>>>> 0.0067790 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1850.151: [GC 1850.151: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2437560 bytes, 2437560 total >>>>>>>> - age 2: 1818240 bytes, 4255800 total >>>>>>>> - age 4: 864 bytes, 4256664 total >>>>>>>> : 106748K->4189K(118016K), 0.0085440 secs] >>>>>>>> 1106630K->1004070K(2084096K), >>>>>>>> 0.0086730 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1850.303: [GC 1850.303: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2408 bytes, 2408 total >>>>>>>> - age 2: 1848744 bytes, 1851152 total >>>>>>>> - age 3: 1818208 bytes, 3669360 total >>>>>>>> : 109149K->3644K(118016K), 0.0072780 secs] >>>>>>>> 1109030K->1003527K(2084096K), >>>>>>>> 0.0073990 secs] [Times: user=0.05 sys=0.00, real=0.00 secs] >>>>>>>> 1850.459: [GC 1850.459: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6760 bytes, 6760 total >>>>>>>> - age 3: 1848744 bytes, 1855504 total >>>>>>>> - age 4: 1818208 bytes, 3673712 total >>>>>>>> : 108604K->3663K(118016K), 0.0080060 secs] >>>>>>>> 1108487K->1003546K(2084096K), >>>>>>>> 0.0081320 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] >>>>>>>> 1850.468: [GC 1850.468: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 296 bytes, 296 total >>>>>>>> - age 2: 5048 bytes, 5344 total >>>>>>>> - age 4: 1848744 bytes, 1854088 total >>>>>>>> : 4093K->1921K(118016K), 0.0078440 secs] >>>>>>>> 1003976K->1003579K(2084096K), >>>>>>>> 0.0079560 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1850.620: [GC 1850.620: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4088 bytes, 4088 total >>>>>>>> : 106881K->17K(118016K), 0.0087130 secs] >>>>>>>> 1108539K->1002911K(2084096K), >>>>>>>> 0.0088300 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1850.777: [GC 1850.777: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 7160 bytes, 7160 total >>>>>>>> : 104977K->14K(118016K), 0.0077920 secs] >>>>>>>> 1107871K->1002907K(2084096K), >>>>>>>> 0.0079110 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1850.928: [GC 1850.928: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6216 bytes, 6216 total >>>>>>>> - age 2: 1056 bytes, 7272 total >>>>>>>> : 104974K->17K(118016K), 0.0062780 secs] >>>>>>>> 1107867K->1002910K(2084096K), >>>>>>>> 0.0064020 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1851.078: [GC 1851.078: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 12168 bytes, 12168 total >>>>>>>> - age 3: 1056 bytes, 13224 total >>>>>>>> : 104977K->20K(118016K), 0.0071340 secs] >>>>>>>> 1107870K->1002913K(2084096K), >>>>>>>> 0.0072520 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1851.229: [GC 1851.229: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6128 bytes, 6128 total >>>>>>>> - age 4: 1056 bytes, 7184 total >>>>>>>> : 104980K->18K(118016K), 0.0071200 secs] >>>>>>>> 1107873K->1002911K(2084096K), >>>>>>>> 0.0072370 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1851.385: [GC 1851.385: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3920 bytes, 3920 total >>>>>>>> : 104978K->14K(118016K), 0.0071250 secs] >>>>>>>> 1107871K->1002908K(2084096K), >>>>>>>> 0.0072340 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1851.535: [GC 1851.535: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2928 bytes, 2928 total >>>>>>>> : 104974K->12K(118016K), 0.0069450 secs] >>>>>>>> 1107868K->1002906K(2084096K), >>>>>>>> 0.0070570 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1851.685: [GC 1851.685: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4080 bytes, 4080 total >>>>>>>> : 104972K->12K(118016K), 0.0068070 secs] >>>>>>>> 1107866K->1002906K(2084096K), >>>>>>>> 0.0069140 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1851.835: [GC 1851.835: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6600 bytes, 6600 total >>>>>>>> : 104972K->12K(118016K), 0.0050100 secs] >>>>>>>> 1107866K->1002906K(2084096K), >>>>>>>> 0.0051180 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1851.989: [GC 1851.989: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4952 bytes, 4952 total >>>>>>>> : 104972K->18K(118016K), 0.0069940 secs] >>>>>>>> 1107866K->1002912K(2084096K), >>>>>>>> 0.0071040 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1852.139: [GC 1852.139: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5416 bytes, 5416 total >>>>>>>> : 104978K->13K(118016K), 0.0073640 secs] >>>>>>>> 1107872K->1002907K(2084096K), >>>>>>>> 0.0074720 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1852.289: [GC 1852.289: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5512 bytes, 5512 total >>>>>>>> - age 2: 1072 bytes, 6584 total >>>>>>>> : 104973K->16K(118016K), 0.0069670 secs] >>>>>>>> 1107867K->1002910K(2084096K), >>>>>>>> 0.0070760 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1852.439: [GC 1852.439: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1928 bytes, 1928 total >>>>>>>> - age 3: 1072 bytes, 3000 total >>>>>>>> : 104976K->9K(118016K), 0.0068990 secs] >>>>>>>> 1107870K->1002903K(2084096K), >>>>>>>> 0.0070040 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1852.446: [GC 1852.446: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2440 bytes, 2440 total >>>>>>>> - age 2: 1672 bytes, 4112 total >>>>>>>> - age 4: 1072 bytes, 5184 total >>>>>>>> : 665K->10K(118016K), 0.0064950 secs] 1003560K->1002904K(2084096K), >>>>>>>> 0.0065840 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1852.596: [GC 1852.596: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 7960 bytes, 7960 total >>>>>>>> : 104970K->14K(118016K), 0.0069390 secs] >>>>>>>> 1107864K->1002909K(2084096K), >>>>>>>> 0.0070450 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1852.746: [GC 1852.746: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6192 bytes, 6192 total >>>>>>>> : 104974K->14K(118016K), 0.0070190 secs] >>>>>>>> 1107869K->1002909K(2084096K), >>>>>>>> 0.0071270 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1852.896: [GC 1852.896: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4744 bytes, 4744 total >>>>>>>> : 104974K->12K(118016K), 0.0068960 secs] >>>>>>>> 1107869K->1002907K(2084096K), >>>>>>>> 0.0070030 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1853.045: [GC 1853.045: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6448 bytes, 6448 total >>>>>>>> : 104972K->16K(118016K), 0.0070700 secs] >>>>>>>> 1107867K->1002911K(2084096K), >>>>>>>> 0.0071760 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1853.194: [GC 1853.195: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5152 bytes, 5152 total >>>>>>>> : 104976K->15K(118016K), 0.0073300 secs] >>>>>>>> 1107871K->1002910K(2084096K), >>>>>>>> 0.0074770 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1853.343: [GC 1853.343: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 7288 bytes, 7288 total >>>>>>>> - age 2: 2336 bytes, 9624 total >>>>>>>> : 104975K->15K(118016K), 0.0076530 secs] >>>>>>>> 1107870K->1002910K(2084096K), >>>>>>>> 0.0077630 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1853.495: [GC 1853.495: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 9192 bytes, 9192 total >>>>>>>> - age 3: 2336 bytes, 11528 total >>>>>>>> : 104975K->21K(118016K), 0.0073360 secs] >>>>>>>> 1107870K->1002916K(2084096K), >>>>>>>> 0.0074490 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1853.648: [GC 1853.648: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 7248 bytes, 7248 total >>>>>>>> - age 2: 2088 bytes, 9336 total >>>>>>>> - age 4: 2336 bytes, 11672 total >>>>>>>> : 104981K->20K(118016K), 0.0050230 secs] >>>>>>>> 1107876K->1002915K(2084096K), >>>>>>>> 0.0051340 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1853.795: [GC 1853.795: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 16744 bytes, 16744 total >>>>>>>> : 104980K->23K(118016K), 0.0070790 secs] >>>>>>>> 1107875K->1002921K(2084096K), >>>>>>>> 0.0071890 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1853.946: [GC 1853.946: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 10768 bytes, 10768 total >>>>>>>> - age 2: 5648 bytes, 16416 total >>>>>>>> : 104983K->25K(118016K), 0.0078080 secs] >>>>>>>> 1107881K->1002923K(2084096K), >>>>>>>> 0.0079190 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1854.100: [GC 1854.100: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4288 bytes, 4288 total >>>>>>>> - age 3: 5648 bytes, 9936 total >>>>>>>> : 104985K->15K(118016K), 0.0074630 secs] >>>>>>>> 1107883K->1002913K(2084096K), >>>>>>>> 0.0075730 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1854.262: [GC 1854.262: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4864 bytes, 4864 total >>>>>>>> - age 4: 5648 bytes, 10512 total >>>>>>>> : 104975K->19K(118016K), 0.0066560 secs] >>>>>>>> 1107873K->1002917K(2084096K), >>>>>>>> 0.0067630 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1854.472: [GC 1854.472: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 44472 bytes, 44472 total >>>>>>>> - age 2: 40 bytes, 44512 total >>>>>>>> : 104979K->53K(118016K), 0.0077930 secs] >>>>>>>> 1107877K->1002956K(2084096K), >>>>>>>> 0.0079010 secs] [Times: user=0.04 sys=0.01, real=0.00 secs] >>>>>>>> 1854.681: [GC 1854.681: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 22872 bytes, 22872 total >>>>>>>> - age 2: 3392 bytes, 26264 total >>>>>>>> - age 3: 40 bytes, 26304 total >>>>>>>> : 105013K->43K(118016K), 0.0076690 secs] >>>>>>>> 1107916K->1002946K(2084096K), >>>>>>>> 0.0077780 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1854.893: [GC 1854.893: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3912 bytes, 3912 total >>>>>>>> - age 2: 19648 bytes, 23560 total >>>>>>>> - age 3: 3392 bytes, 26952 total >>>>>>>> - age 4: 40 bytes, 26992 total >>>>>>>> : 105003K->37K(118016K), 0.0073320 secs] >>>>>>>> 1107906K->1002940K(2084096K), >>>>>>>> 0.0074370 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1855.105: [GC 1855.105: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4120 bytes, 4120 total >>>>>>>> - age 3: 19648 bytes, 23768 total >>>>>>>> - age 4: 3392 bytes, 27160 total >>>>>>>> : 104997K->37K(118016K), 0.0067290 secs] >>>>>>>> 1107900K->1002940K(2084096K), >>>>>>>> 0.0068370 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1855.316: [GC 1855.316: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4280 bytes, 4280 total >>>>>>>> - age 4: 19648 bytes, 23928 total >>>>>>>> : 104997K->33K(118016K), 0.0069810 secs] >>>>>>>> 1107900K->1002940K(2084096K), >>>>>>>> 0.0070890 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1855.527: [GC 1855.527: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3992 bytes, 3992 total >>>>>>>> : 104993K->6K(118016K), 0.0050780 secs] >>>>>>>> 1107900K->1002932K(2084096K), >>>>>>>> 0.0051890 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1855.738: [GC 1855.738: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 9648 bytes, 9648 total >>>>>>>> : 104966K->18K(118016K), 0.0066190 secs] >>>>>>>> 1107892K->1002943K(2084096K), >>>>>>>> 0.0067290 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1855.948: [GC 1855.949: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 7208 bytes, 7208 total >>>>>>>> : 104978K->15K(118016K), 0.0067150 secs] >>>>>>>> 1107903K->1002941K(2084096K), >>>>>>>> 0.0068250 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1856.160: [GC 1856.160: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5816 bytes, 5816 total >>>>>>>> - age 2: 1584 bytes, 7400 total >>>>>>>> : 104975K->13K(118016K), 0.0070440 secs] >>>>>>>> 1107901K->1002939K(2084096K), >>>>>>>> 0.0071540 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1856.372: [GC 1856.372: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 888 bytes, 888 total >>>>>>>> - age 3: 1584 bytes, 2472 total >>>>>>>> : 104973K->13K(118016K), 0.0071500 secs] >>>>>>>> 1107899K->1002939K(2084096K), >>>>>>>> 0.0072580 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1856.584: [GC 1856.584: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2960 bytes, 2960 total >>>>>>>> - age 4: 1584 bytes, 4544 total >>>>>>>> : 104973K->11K(118016K), 0.0061030 secs] >>>>>>>> 1107899K->1002937K(2084096K), >>>>>>>> 0.0062140 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1856.794: [GC 1856.795: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2592 bytes, 2592 total >>>>>>>> : 104971K->8K(118016K), 0.0066810 secs] >>>>>>>> 1107897K->1002935K(2084096K), >>>>>>>> 0.0067900 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] >>>>>>>> 1857.008: [GC 1857.008: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3400 bytes, 3400 total >>>>>>>> : 104968K->10K(118016K), 0.0062730 secs] >>>>>>>> 1107895K->1002937K(2084096K), >>>>>>>> 0.0063790 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1857.219: [GC 1857.219: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2336 bytes, 2336 total >>>>>>>> : 104970K->8K(118016K), 0.0064340 secs] >>>>>>>> 1107897K->1002935K(2084096K), >>>>>>>> 0.0065380 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1857.432: [GC 1857.432: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1920 bytes, 1920 total >>>>>>>> : 104968K->8K(118016K), 0.0069200 secs] >>>>>>>> 1107895K->1002935K(2084096K), >>>>>>>> 0.0070300 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1857.646: [GC 1857.646: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6064 bytes, 6064 total >>>>>>>> : 104968K->14K(118016K), 0.0060630 secs] >>>>>>>> 1107895K->1002941K(2084096K), >>>>>>>> 0.0061700 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1857.868: [GC 1857.868: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5392 bytes, 5392 total >>>>>>>> - age 2: 2064 bytes, 7456 total >>>>>>>> : 104974K->14K(118016K), 0.0074520 secs] >>>>>>>> 1107901K->1002941K(2084096K), >>>>>>>> 0.0075600 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1858.090: [GC 1858.090: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 7456 bytes, 7456 total >>>>>>>> - age 3: 2064 bytes, 9520 total >>>>>>>> : 104974K->18K(118016K), 0.0048920 secs] >>>>>>>> 1107901K->1002945K(2084096K), >>>>>>>> 0.0050280 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1858.314: [GC 1858.314: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 13152 bytes, 13152 total >>>>>>>> - age 4: 2064 bytes, 15216 total >>>>>>>> : 104978K->23K(118016K), 0.0060380 secs] >>>>>>>> 1107905K->1002950K(2084096K), >>>>>>>> 0.0061430 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1858.539: [GC 1858.539: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 9632 bytes, 9632 total >>>>>>>> - age 2: 5648 bytes, 15280 total >>>>>>>> : 104983K->19K(118016K), 0.0068210 secs] >>>>>>>> 1107910K->1002948K(2084096K), >>>>>>>> 0.0069270 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1864.962: [GC 1864.962: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2905592 bytes, 2905592 total >>>>>>>> - age 3: 5648 bytes, 2911240 total >>>>>>>> : 104979K->2850K(118016K), 0.0075130 secs] >>>>>>>> 1107908K->1005779K(2084096K), >>>>>>>> 0.0076360 secs] [Times: user=0.04 sys=0.01, real=0.00 secs] >>>>>>>> 1865.422: [GC 1865.422: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4040 bytes, 4040 total >>>>>>>> - age 2: 2902672 bytes, 2906712 total >>>>>>>> - age 4: 5648 bytes, 2912360 total >>>>>>>> : 107810K->2901K(118016K), 0.0069870 secs] >>>>>>>> 1110739K->1005830K(2084096K), >>>>>>>> 0.0070950 secs] [Times: user=0.04 sys=0.01, real=0.00 secs] >>>>>>>> 1865.878: [GC 1865.878: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 416 bytes, 416 total >>>>>>>> - age 2: 3512 bytes, 3928 total >>>>>>>> - age 3: 2902672 bytes, 2906600 total >>>>>>>> : 107861K->2932K(118016K), 0.0066050 secs] >>>>>>>> 1110790K->1005867K(2084096K), >>>>>>>> 0.0067270 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1866.328: [GC 1866.328: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 640 bytes, 640 total >>>>>>>> - age 3: 3512 bytes, 4152 total >>>>>>>> - age 4: 2902672 bytes, 2906824 total >>>>>>>> : 107892K->2976K(118016K), 0.0069830 secs] >>>>>>>> 1110827K->1005911K(2084096K), >>>>>>>> 0.0070990 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1866.799: [GC 1866.799: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 72 bytes, 72 total >>>>>>>> - age 4: 3512 bytes, 3584 total >>>>>>>> : 107936K->32K(118016K), 0.0077140 secs] >>>>>>>> 1110871K->1005802K(2084096K), >>>>>>>> 0.0078210 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] >>>>>>>> 1867.253: [GC 1867.253: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2032 bytes, 2032 total >>>>>>>> : 104992K->7K(118016K), 0.0067320 secs] >>>>>>>> 1110762K->1005780K(2084096K), >>>>>>>> 0.0068360 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1867.726: [GC 1867.726: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 640 bytes, 640 total >>>>>>>> : 104967K->4K(118016K), 0.0067880 secs] >>>>>>>> 1110740K->1005777K(2084096K), >>>>>>>> 0.0068990 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1868.197: [GC 1868.198: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1704 bytes, 1704 total >>>>>>>> : 104964K->8K(118016K), 0.0058240 secs] >>>>>>>> 1110737K->1005781K(2084096K), >>>>>>>> 0.0059360 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1868.689: [GC 1868.689: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2360 bytes, 2360 total >>>>>>>> : 104968K->8K(118016K), 0.0058100 secs] >>>>>>>> 1110741K->1005781K(2084096K), >>>>>>>> 0.0059260 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1869.160: [GC 1869.160: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1400 bytes, 1400 total >>>>>>>> : 104968K->6K(118016K), 0.0068580 secs] >>>>>>>> 1110741K->1005779K(2084096K), >>>>>>>> 0.0069720 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1869.633: [GC 1869.633: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2944 bytes, 2944 total >>>>>>>> : 104966K->8K(118016K), 0.0066850 secs] >>>>>>>> 1110739K->1005781K(2084096K), >>>>>>>> 0.0067980 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1870.106: [GC 1870.106: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2144 bytes, 2144 total >>>>>>>> : 104968K->6K(118016K), 0.0073740 secs] >>>>>>>> 1110741K->1005779K(2084096K), >>>>>>>> 0.0074960 secs] [Times: user=0.02 sys=0.01, real=0.01 secs] >>>>>>>> 1870.578: [GC 1870.578: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1128 bytes, 1128 total >>>>>>>> : 104966K->8K(118016K), 0.0066560 secs] >>>>>>>> 1110739K->1005781K(2084096K), >>>>>>>> 0.0067600 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1871.049: [GC 1871.049: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 640 bytes, 640 total >>>>>>>> : 104968K->6K(118016K), 0.0060350 secs] >>>>>>>> 1110741K->1005779K(2084096K), >>>>>>>> 0.0061360 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1871.528: [GC 1871.528: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 10336 bytes, 10336 total >>>>>>>> : 104966K->15K(118016K), 0.0064930 secs] >>>>>>>> 1110739K->1005788K(2084096K), >>>>>>>> 0.0066020 secs] [Times: user=0.02 sys=0.01, real=0.01 secs] >>>>>>>> 1873.405: [GC 1873.406: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3680280 bytes, 3680280 total >>>>>>>> - age 2: 5248 bytes, 3685528 total >>>>>>>> : 104881K->3608K(118016K), 0.0073650 secs] >>>>>>>> 1110654K->1009381K(2084096K), >>>>>>>> 0.0074970 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1873.760: [GC 1873.761: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1586848 bytes, 1586848 total >>>>>>>> - age 2: 3679432 bytes, 5266280 total >>>>>>>> - age 3: 5248 bytes, 5271528 total >>>>>>>> : 108568K->5210K(118016K), 0.0079460 secs] >>>>>>>> 1114341K->1010983K(2084096K), >>>>>>>> 0.0080830 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] >>>>>>>> 1874.004: [GC 1874.004: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4312 bytes, 4312 total >>>>>>>> - age 2: 1582528 bytes, 1586840 total >>>>>>>> - age 3: 3679432 bytes, 5266272 total >>>>>>>> - age 4: 5248 bytes, 5271520 total >>>>>>>> : 110170K->5385K(118016K), 0.0084690 secs] >>>>>>>> 1115943K->1011158K(2084096K), >>>>>>>> 0.0086300 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1874.240: [GC 1874.240: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3344 bytes, 3344 total >>>>>>>> - age 2: 3512 bytes, 6856 total >>>>>>>> - age 3: 1582528 bytes, 1589384 total >>>>>>>> - age 4: 3679432 bytes, 5268816 total >>>>>>>> : 110345K->5388K(118016K), 0.0092950 secs] >>>>>>>> 1116118K->1011167K(2084096K), >>>>>>>> 0.0094130 secs] [Times: user=0.06 sys=0.01, real=0.01 secs] >>>>>>>> 1874.479: [GC 1874.479: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5536 bytes, 5536 total >>>>>>>> - age 3: 3512 bytes, 9048 total >>>>>>>> - age 4: 1581112 bytes, 1590160 total >>>>>>>> : 110348K->1843K(118016K), 0.0130380 secs] >>>>>>>> 1116127K->1011214K(2084096K), >>>>>>>> 0.0131610 secs] [Times: user=0.06 sys=0.01, real=0.02 secs] >>>>>>>> 1874.720: [GC 1874.720: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3744 bytes, 3744 total >>>>>>>> - age 2: 1600 bytes, 5344 total >>>>>>>> - age 4: 3512 bytes, 8856 total >>>>>>>> : 106803K->56K(118016K), 0.0063140 secs] >>>>>>>> 1116174K->1010972K(2084096K), >>>>>>>> 0.0064280 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1874.955: [GC 1874.955: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3096 bytes, 3096 total >>>>>>>> - age 3: 1600 bytes, 4696 total >>>>>>>> : 105016K->16K(118016K), 0.0073310 secs] >>>>>>>> 1115932K->1010935K(2084096K), >>>>>>>> 0.0074390 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1875.191: [GC 1875.191: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1472 bytes, 1472 total >>>>>>>> - age 4: 1600 bytes, 3072 total >>>>>>>> : 104976K->9K(118016K), 0.0059430 secs] >>>>>>>> 1115895K->1010928K(2084096K), >>>>>>>> 0.0060450 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1875.427: [GC 1875.427: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1376 bytes, 1376 total >>>>>>>> : 104969K->8K(118016K), 0.0065520 secs] >>>>>>>> 1115888K->1010928K(2084096K), >>>>>>>> 0.0066600 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1875.664: [GC 1875.664: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1424 bytes, 1424 total >>>>>>>> : 104968K->10K(118016K), 0.0063910 secs] >>>>>>>> 1115888K->1010930K(2084096K), >>>>>>>> 0.0064950 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1875.900: [GC 1875.900: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6824 bytes, 6824 total >>>>>>>> : 104970K->18K(118016K), 0.0057500 secs] >>>>>>>> 1115890K->1010938K(2084096K), >>>>>>>> 0.0058550 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1876.135: [GC 1876.135: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2200 bytes, 2200 total >>>>>>>> : 104978K->8K(118016K), 0.0069470 secs] >>>>>>>> 1115898K->1010928K(2084096K), >>>>>>>> 0.0070510 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1876.372: [GC 1876.372: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2192 bytes, 2192 total >>>>>>>> : 104968K->10K(118016K), 0.0070880 secs] >>>>>>>> 1115888K->1010930K(2084096K), >>>>>>>> 0.0071870 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] >>>>>>>> 1876.609: [GC 1876.609: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1104 bytes, 1104 total >>>>>>>> : 104970K->10K(118016K), 0.0066470 secs] >>>>>>>> 1115890K->1010930K(2084096K), >>>>>>>> 0.0067550 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1876.845: [GC 1876.845: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5312 bytes, 5312 total >>>>>>>> : 104970K->14K(118016K), 0.0062250 secs] >>>>>>>> 1115890K->1010934K(2084096K), >>>>>>>> 0.0063320 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1877.080: [GC 1877.080: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1736 bytes, 1736 total >>>>>>>> : 104974K->10K(118016K), 0.0056870 secs] >>>>>>>> 1115894K->1010930K(2084096K), >>>>>>>> 0.0057890 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1877.315: [GC 1877.315: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4440 bytes, 4440 total >>>>>>>> : 104970K->10K(118016K), 0.0066800 secs] >>>>>>>> 1115890K->1010930K(2084096K), >>>>>>>> 0.0067890 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1877.552: [GC 1877.552: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 7456 bytes, 7456 total >>>>>>>> : 104970K->12K(118016K), 0.0071600 secs] >>>>>>>> 1115890K->1010932K(2084096K), >>>>>>>> 0.0072620 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1877.788: [GC 1877.788: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1928 bytes, 1928 total >>>>>>>> : 104972K->8K(118016K), 0.0065670 secs] >>>>>>>> 1115892K->1010928K(2084096K), >>>>>>>> 0.0066730 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1878.024: [GC 1878.024: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3000 bytes, 3000 total >>>>>>>> : 104968K->10K(118016K), 0.0067640 secs] >>>>>>>> 1115888K->1010930K(2084096K), >>>>>>>> 0.0068670 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1878.259: [GC 1878.259: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6656 bytes, 6656 total >>>>>>>> : 104970K->14K(118016K), 0.0072400 secs] >>>>>>>> 1115890K->1010934K(2084096K), >>>>>>>> 0.0073450 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1878.497: [GC 1878.497: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3088 bytes, 3088 total >>>>>>>> : 104974K->12K(118016K), 0.0066530 secs] >>>>>>>> 1115894K->1010932K(2084096K), >>>>>>>> 0.0067550 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1878.734: [GC 1878.734: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 10048 bytes, 10048 total >>>>>>>> : 104972K->16K(118016K), 0.0065520 secs] >>>>>>>> 1115892K->1010936K(2084096K), >>>>>>>> 0.0066680 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1878.971: [GC 1878.971: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 9688 bytes, 9688 total >>>>>>>> : 104976K->18K(118016K), 0.0056070 secs] >>>>>>>> 1115896K->1010938K(2084096K), >>>>>>>> 0.0057230 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1879.207: [GC 1879.207: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3208 bytes, 3208 total >>>>>>>> : 104978K->10K(118016K), 0.0057250 secs] >>>>>>>> 1115898K->1010930K(2084096K), >>>>>>>> 0.0058360 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1879.442: [GC 1879.442: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3272 bytes, 3272 total >>>>>>>> : 104970K->8K(118016K), 0.0062510 secs] >>>>>>>> 1115890K->1010928K(2084096K), >>>>>>>> 0.0063680 secs] [Times: user=0.02 sys=0.01, real=0.00 secs] >>>>>>>> 1879.678: [GC 1879.679: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3224 bytes, 3224 total >>>>>>>> : 104968K->10K(118016K), 0.0057810 secs] >>>>>>>> 1115888K->1010930K(2084096K), >>>>>>>> 0.0058990 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1879.914: [GC 1879.914: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3160 bytes, 3160 total >>>>>>>> : 104970K->8K(118016K), 0.0064760 secs] >>>>>>>> 1115890K->1010928K(2084096K), >>>>>>>> 0.0065950 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1880.148: [GC 1880.148: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3080 bytes, 3080 total >>>>>>>> : 104968K->12K(118016K), 0.0063810 secs] >>>>>>>> 1115888K->1010932K(2084096K), >>>>>>>> 0.0064960 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1880.390: [GC 1880.390: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 17512 bytes, 17512 total >>>>>>>> : 104972K->24K(118016K), 0.0056060 secs] >>>>>>>> 1115892K->1010944K(2084096K), >>>>>>>> 0.0057250 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1880.634: [GC 1880.634: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 14640 bytes, 14640 total >>>>>>>> : 104984K->22K(118016K), 0.0062140 secs] >>>>>>>> 1115904K->1010943K(2084096K), >>>>>>>> 0.0063360 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1880.880: [GC 1880.880: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2072 bytes, 2072 total >>>>>>>> - age 2: 10896 bytes, 12968 total >>>>>>>> : 104982K->23K(118016K), 0.0048630 secs] >>>>>>>> 1115903K->1010943K(2084096K), >>>>>>>> 0.0050290 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1888.172: [GC 1888.172: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2445208 bytes, 2445208 total >>>>>>>> - age 2: 272 bytes, 2445480 total >>>>>>>> - age 3: 10896 bytes, 2456376 total >>>>>>>> : 104983K->2405K(118016K), 0.0073800 secs] >>>>>>>> 1115903K->1013325K(2084096K), >>>>>>>> 0.0074960 secs] [Times: user=0.05 sys=0.01, real=0.00 secs] >>>>>>>> 1888.643: [GC 1888.643: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3784 bytes, 3784 total >>>>>>>> - age 2: 2439992 bytes, 2443776 total >>>>>>>> - age 3: 272 bytes, 2444048 total >>>>>>>> - age 4: 10896 bytes, 2454944 total >>>>>>>> : 107365K->2457K(118016K), 0.0071510 secs] >>>>>>>> 1118285K->1013377K(2084096K), >>>>>>>> 0.0072550 secs] [Times: user=0.05 sys=0.00, real=0.00 secs] >>>>>>>> 1889.109: [GC 1889.109: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 8016 bytes, 8016 total >>>>>>>> - age 3: 2439992 bytes, 2448008 total >>>>>>>> - age 4: 272 bytes, 2448280 total >>>>>>>> : 107417K->2560K(118016K), 0.0070890 secs] >>>>>>>> 1118337K->1013491K(2084096K), >>>>>>>> 0.0071980 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1889.579: [GC 1889.579: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2384 bytes, 2384 total >>>>>>>> - age 2: 1600 bytes, 3984 total >>>>>>>> - age 4: 2439992 bytes, 2443976 total >>>>>>>> : 107520K->2545K(118016K), 0.0071830 secs] >>>>>>>> 1118451K->1013476K(2084096K), >>>>>>>> 0.0072940 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] >>>>>>>> 1890.045: [GC 1890.045: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 880 bytes, 880 total >>>>>>>> - age 3: 1600 bytes, 2480 total >>>>>>>> : 107505K->166K(118016K), 0.0074170 secs] >>>>>>>> 1118436K->1013480K(2084096K), >>>>>>>> 0.0075280 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] >>>>>>>> 1890.515: [GC 1890.515: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 128 bytes, 128 total >>>>>>>> - age 2: 96 bytes, 224 total >>>>>>>> - age 4: 1600 bytes, 1824 total >>>>>>>> : 105126K->33K(118016K), 0.0066310 secs] >>>>>>>> 1118440K->1013347K(2084096K), >>>>>>>> 0.0067340 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1890.984: [GC 1890.984: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1176 bytes, 1176 total >>>>>>>> - age 3: 96 bytes, 1272 total >>>>>>>> : 104993K->8K(118016K), 0.0069150 secs] >>>>>>>> 1118307K->1013324K(2084096K), >>>>>>>> 0.0070120 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1891.452: [GC 1891.452: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 320 bytes, 320 total >>>>>>>> - age 4: 96 bytes, 416 total >>>>>>>> : 104968K->6K(118016K), 0.0061500 secs] >>>>>>>> 1118284K->1013321K(2084096K), >>>>>>>> 0.0062530 secs] [Times: user=0.02 sys=0.01, real=0.00 secs] >>>>>>>> 1891.922: [GC 1891.922: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1264 bytes, 1264 total >>>>>>>> : 104966K->6K(118016K), 0.0065900 secs] >>>>>>>> 1118281K->1013321K(2084096K), >>>>>>>> 0.0066900 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] >>>>>>>> 1892.390: [GC 1892.390: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2984 bytes, 2984 total >>>>>>>> : 104966K->8K(118016K), 0.0062690 secs] >>>>>>>> 1118281K->1013323K(2084096K), >>>>>>>> 0.0063680 secs] [Times: user=0.02 sys=0.01, real=0.01 secs] >>>>>>>> 1892.859: [GC 1892.859: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1520 bytes, 1520 total >>>>>>>> : 104968K->8K(118016K), 0.0060840 secs] >>>>>>>> 1118283K->1013323K(2084096K), >>>>>>>> 0.0061810 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1893.328: [GC 1893.328: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1040 bytes, 1040 total >>>>>>>> : 104968K->4K(118016K), 0.0062090 secs] >>>>>>>> 1118283K->1013319K(2084096K), >>>>>>>> 0.0063120 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1893.795: [GC 1893.795: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1224 bytes, 1224 total >>>>>>>> : 104964K->6K(118016K), 0.0061420 secs] >>>>>>>> 1118279K->1013321K(2084096K), >>>>>>>> 0.0062410 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> 1894.265: [GC 1894.265: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2080 bytes, 2080 total >>>>>>>> : 104966K->4K(118016K), 0.0062520 secs] >>>>>>>> 1118281K->1013319K(2084096K), >>>>>>>> 0.0063550 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] >>>>>>>> 1894.731: [GC 1894.731: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1936 bytes, 1936 total >>>>>>>> : 104964K->6K(118016K), 0.0058280 secs] >>>>>>>> 1118279K->1013321K(2084096K), >>>>>>>> 0.0059240 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1895.227: [GC 1895.227: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 6928 bytes, 6928 total >>>>>>>> : 104966K->9K(118016K), 0.0046570 secs] >>>>>>>> 1118281K->1013325K(2084096K), >>>>>>>> 0.0047560 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1898.054: [GC 1898.054: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3115424 bytes, 3115424 total >>>>>>>> - age 2: 5648 bytes, 3121072 total >>>>>>>> : 104969K->3055K(118016K), 0.0073760 secs] >>>>>>>> 1118285K->1016371K(2084096K), >>>>>>>> 0.0074910 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1898.529: [GC 1898.529: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 169472 bytes, 169472 total >>>>>>>> - age 2: 3113760 bytes, 3283232 total >>>>>>>> - age 3: 5648 bytes, 3288880 total >>>>>>>> : 108015K->3293K(118016K), 0.0079730 secs] >>>>>>>> 1121331K->1016609K(2084096K), >>>>>>>> 0.0080760 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] >>>>>>>> 1898.987: [GC 1898.987: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1784824 bytes, 1784824 total >>>>>>>> - age 2: 165616 bytes, 1950440 total >>>>>>>> - age 3: 3113728 bytes, 5064168 total >>>>>>>> - age 4: 5648 bytes, 5069816 total >>>>>>>> : 108253K->5156K(118016K), 0.0079720 secs] >>>>>>>> 1121569K->1018471K(2084096K), >>>>>>>> 0.0080790 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1899.456: [GC 1899.456: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 632184 bytes, 632184 total >>>>>>>> - age 2: 1774984 bytes, 2407168 total >>>>>>>> - age 3: 165616 bytes, 2572784 total >>>>>>>> - age 4: 2529728 bytes, 5102512 total >>>>>>>> : 110116K->5223K(118016K), 0.0083540 secs] >>>>>>>> 1123431K->1018544K(2084096K), >>>>>>>> 0.0084690 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] >>>>>>>> 1899.928: [GC 1899.928: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 17040 bytes, 17040 total >>>>>>>> - age 2: 632016 bytes, 649056 total >>>>>>>> - age 3: 1774968 bytes, 2424024 total >>>>>>>> - age 4: 165616 bytes, 2589640 total >>>>>>>> : 110183K->2879K(118016K), 0.0081040 secs] >>>>>>>> 1123504K->1018671K(2084096K), >>>>>>>> 0.0082020 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1900.405: [GC 1900.405: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 56064 bytes, 56064 total >>>>>>>> - age 2: 14376 bytes, 70440 total >>>>>>>> - age 3: 631968 bytes, 702408 total >>>>>>>> - age 4: 1774968 bytes, 2477376 total >>>>>>>> : 107839K->2611K(118016K), 0.0078230 secs] >>>>>>>> 1123631K->1018564K(2084096K), >>>>>>>> 0.0079320 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] >>>>>>>> 1900.897: [GC 1900.897: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 40456 bytes, 40456 total >>>>>>>> - age 2: 51056 bytes, 91512 total >>>>>>>> - age 3: 14376 bytes, 105888 total >>>>>>>> - age 4: 631968 bytes, 737856 total >>>>>>>> : 107571K->942K(118016K), 0.0072930 secs] >>>>>>>> 1123524K->1018629K(2084096K), >>>>>>>> 0.0074000 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1901.364: [GC 1901.364: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 67120 bytes, 67120 total >>>>>>>> - age 2: 31192 bytes, 98312 total >>>>>>>> - age 3: 51056 bytes, 149368 total >>>>>>>> - age 4: 14328 bytes, 163696 total >>>>>>>> : 105902K->293K(118016K), 0.0057690 secs] >>>>>>>> 1123589K->1018597K(2084096K), >>>>>>>> 0.0058800 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1901.831: [GC 1901.831: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 37280 bytes, 37280 total >>>>>>>> - age 2: 66384 bytes, 103664 total >>>>>>>> - age 3: 31192 bytes, 134856 total >>>>>>>> - age 4: 51056 bytes, 185912 total >>>>>>>> : 105253K->223K(118016K), 0.0050510 secs] >>>>>>>> 1123557K->1018541K(2084096K), >>>>>>>> 0.0051540 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1902.294: [GC 1902.294: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 105720 bytes, 105720 total >>>>>>>> - age 2: 36544 bytes, 142264 total >>>>>>>> - age 3: 66336 bytes, 208600 total >>>>>>>> - age 4: 31192 bytes, 239792 total >>>>>>>> : 105183K->324K(118016K), 0.0056210 secs] >>>>>>>> 1123501K->1018692K(2084096K), >>>>>>>> 0.0057240 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1902.753: [GC 1902.753: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1567200 bytes, 1567200 total >>>>>>>> - age 2: 105632 bytes, 1672832 total >>>>>>>> - age 3: 36544 bytes, 1709376 total >>>>>>>> - age 4: 66336 bytes, 1775712 total >>>>>>>> : 105284K->1882K(118016K), 0.0082650 secs] >>>>>>>> 1123652K->1020280K(2084096K), >>>>>>>> 0.0083680 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1903.223: [GC 1903.223: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 30040 bytes, 30040 total >>>>>>>> - age 2: 1566096 bytes, 1596136 total >>>>>>>> - age 3: 105632 bytes, 1701768 total >>>>>>>> - age 4: 36544 bytes, 1738312 total >>>>>>>> : 106842K->1884K(118016K), 0.0059610 secs] >>>>>>>> 1125240K->1020347K(2084096K), >>>>>>>> 0.0060690 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1903.693: [GC 1903.693: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 8240 bytes, 8240 total >>>>>>>> - age 2: 28048 bytes, 36288 total >>>>>>>> - age 3: 1566096 bytes, 1602384 total >>>>>>>> - age 4: 105632 bytes, 1708016 total >>>>>>>> : 106844K->1789K(118016K), 0.0059920 secs] >>>>>>>> 1125307K->1020288K(2084096K), >>>>>>>> 0.0060920 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1904.164: [GC 1904.164: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 460872 bytes, 460872 total >>>>>>>> - age 2: 6264 bytes, 467136 total >>>>>>>> - age 3: 27960 bytes, 495096 total >>>>>>>> - age 4: 1566096 bytes, 2061192 total >>>>>>>> : 106749K->2165K(118016K), 0.0062860 secs] >>>>>>>> 1125248K->1020767K(2084096K), >>>>>>>> 0.0063890 secs] [Times: user=0.04 sys=0.01, real=0.00 secs] >>>>>>>> 1904.624: [GC 1904.624: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2709336 bytes, 2709336 total >>>>>>>> - age 2: 458392 bytes, 3167728 total >>>>>>>> - age 3: 6168 bytes, 3173896 total >>>>>>>> - age 4: 27960 bytes, 3201856 total >>>>>>>> : 107125K->3170K(118016K), 0.0083420 secs] >>>>>>>> 1125727K->1023301K(2084096K), >>>>>>>> 0.0084480 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1905.094: [GC 1905.094: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 659712 bytes, 659712 total >>>>>>>> - age 2: 2228256 bytes, 2887968 total >>>>>>>> - age 3: 458392 bytes, 3346360 total >>>>>>>> - age 4: 6168 bytes, 3352528 total >>>>>>>> : 108130K->3375K(118016K), 0.0085350 secs] >>>>>>>> 1128261K->1023534K(2084096K), >>>>>>>> 0.0086420 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] >>>>>>>> 1905.505: [GC 1905.505: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 807096 bytes, 807096 total >>>>>>>> - age 2: 657808 bytes, 1464904 total >>>>>>>> - age 3: 2227512 bytes, 3692416 total >>>>>>>> - age 4: 458392 bytes, 4150808 total >>>>>>>> : 108335K->4146K(118016K), 0.0082600 secs] >>>>>>>> 1128494K->1024310K(2084096K), >>>>>>>> 0.0083750 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] >>>>>>>> 1905.749: [GC 1905.749: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3712 bytes, 3712 total >>>>>>>> - age 2: 802912 bytes, 806624 total >>>>>>>> - age 3: 657808 bytes, 1464432 total >>>>>>>> - age 4: 2227512 bytes, 3691944 total >>>>>>>> : 109106K->3774K(118016K), 0.0088880 secs] >>>>>>>> 1129270K->1024387K(2084096K), >>>>>>>> 0.0090000 secs] [Times: user=0.06 sys=0.01, real=0.01 secs] >>>>>>>> tfd.totalHits=0, records wanted=290 >>>>>>>> qryStr=( S000:"MILTARY SYTEMS ENGINEER" ) AND >>>>>>>> (TEAMID0000000022:0000000022 >>>>>>>> OR TEAMID0000000000:0000000000) >>>>>>>> LI [380 at /lucene1/index/]-26-Mar 06:34:25-Found hits=0, rset=0, >>>>>>>> has >>>>>>>> more=false >>>>>>>> LI [380 at /lucene1/index/]-26-Mar 06:34:25-Time--Total=1010, >>>>>>>> query=1010, >>>>>>>> fetch=0 >>>>>>>> 1905.993: [GC 1905.993: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3232 bytes, 3232 total >>>>>>>> - age 3: 792792 bytes, 796024 total >>>>>>>> - age 4: 638624 bytes, 1434648 total >>>>>>>> : 108734K->1734K(118016K), 0.0087890 secs] >>>>>>>> 1129347K->1024522K(2084096K), >>>>>>>> 0.0089000 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] >>>>>>>> 1906.249: [GC 1906.249: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2984 bytes, 2984 total >>>>>>>> - age 2: 1424 bytes, 4408 total >>>>>>>> - age 4: 792792 bytes, 797200 total >>>>>>>> : 106694K->947K(118016K), 0.0062530 secs] >>>>>>>> 1129482K->1024358K(2084096K), >>>>>>>> 0.0063540 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 602304 bytes, 602304 total >>>>>>>> - age 2: 1992 bytes, 604296 total >>>>>>>> - age 3: 1392 bytes, 605688 total >>>>>>>> : 105907K->706K(118016K), 0.0064420 secs] >>>>>>>> 1129318K->1024892K(2084096K), >>>>>>>> 0.0065540 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] >>>>>>>> 1906.757: [GC 1906.757: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 9480 bytes, 9480 total >>>>>>>> - age 2: 598512 bytes, 607992 total >>>>>>>> - age 3: 1992 bytes, 609984 total >>>>>>>> - age 4: 1392 bytes, 611376 total >>>>>>>> : 105666K->633K(118016K), 0.0060690 secs] >>>>>>>> 1129852K->1024818K(2084096K), >>>>>>>> 0.0061710 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1907.134: [GC 1907.134: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 5192 bytes, 5192 total >>>>>>>> - age 2: 4072 bytes, 9264 total >>>>>>>> - age 3: 598512 bytes, 607776 total >>>>>>>> - age 4: 1992 bytes, 609768 total >>>>>>>> : 105593K->614K(118016K), 0.0078790 secs] >>>>>>>> 1129778K->1024801K(2084096K), >>>>>>>> 0.0079830 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1907.618: [GC 1907.619: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 89488 bytes, 89488 total >>>>>>>> - age 2: 2248 bytes, 91736 total >>>>>>>> - age 3: 4024 bytes, 95760 total >>>>>>>> - age 4: 598512 bytes, 694272 total >>>>>>>> : 105574K->699K(118016K), 0.0082250 secs] >>>>>>>> 1129761K->1024888K(2084096K), >>>>>>>> 0.0083310 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1908.089: [GC 1908.089: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 105448 bytes, 105448 total >>>>>>>> - age 2: 2088 bytes, 107536 total >>>>>>>> - age 3: 2152 bytes, 109688 total >>>>>>>> - age 4: 4024 bytes, 113712 total >>>>>>>> : 105659K->185K(118016K), 0.0078770 secs] >>>>>>>> 1129848K->1024958K(2084096K), >>>>>>>> 0.0079850 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] >>>>>>>> 1908.563: [GC 1908.563: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 678656 bytes, 678656 total >>>>>>>> - age 2: 103096 bytes, 781752 total >>>>>>>> - age 3: 2088 bytes, 783840 total >>>>>>>> - age 4: 2104 bytes, 785944 total >>>>>>>> : 105145K->798K(118016K), 0.0058960 secs] >>>>>>>> 1129918K->1025575K(2084096K), >>>>>>>> 0.0060010 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1909.482: [GC 1909.482: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 609888 bytes, 609888 total >>>>>>>> - age 2: 27312 bytes, 637200 total >>>>>>>> - age 3: 629728 bytes, 1266928 total >>>>>>>> - age 4: 102952 bytes, 1369880 total >>>>>>>> : 105766K->1414K(118016K), 0.0059090 secs] >>>>>>>> 1130545K->1026195K(2084096K), >>>>>>>> 0.0060130 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1909.953: [GC 1909.953: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 72 bytes, 72 total >>>>>>>> - age 2: 606424 bytes, 606496 total >>>>>>>> - age 3: 27312 bytes, 633808 total >>>>>>>> - age 4: 629728 bytes, 1263536 total >>>>>>>> : 106374K->1338K(118016K), 0.0058170 secs] >>>>>>>> 1131155K->1026220K(2084096K), >>>>>>>> 0.0059100 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] >>>>>>>> 1910.418: [GC 1910.418: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 1584 bytes, 1584 total >>>>>>>> - age 3: 606424 bytes, 608008 total >>>>>>>> - age 4: 27312 bytes, 635320 total >>>>>>>> : 106298K->752K(118016K), 0.0055870 secs] >>>>>>>> 1131180K->1026248K(2084096K), >>>>>>>> 0.0056800 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1910.885: [GC 1910.885: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 4488 bytes, 4488 total >>>>>>>> - age 4: 606440 bytes, 610928 total >>>>>>>> : 105712K->666K(118016K), 0.0054040 secs] >>>>>>>> 1131208K->1026190K(2084096K), >>>>>>>> 0.0055050 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1911.354: [GC 1911.354: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2976 bytes, 2976 total >>>>>>>> : 105626K->26K(118016K), 0.0075400 secs] >>>>>>>> 1131150K->1026142K(2084096K), >>>>>>>> 0.0076410 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] >>>>>>>> 1911.823: [GC 1911.823: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2160 bytes, 2160 total >>>>>>>> : 104986K->6K(118016K), 0.0073880 secs] >>>>>>>> 1131102K->1026121K(2084096K), >>>>>>>> 0.0074760 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] >>>>>>>> 1912.290: [GC 1912.290: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 2176 bytes, 2176 total >>>>>>>> : 104966K->8K(118016K), 0.0069440 secs] >>>>>>>> 1131081K->1026123K(2084096K), >>>>>>>> 0.0070370 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] >>>>>>>> 1912.771: [GC 1912.771: [ParNew >>>>>>>> Desired survivor size 6684672 bytes, new threshold 4 (max 4) >>>>>>>> - age 1: 3056 bytes, 3056 total >>>>>>>> : 104968K->10K(118016K), 0.0071230 secs] >>>>>>>> 1131083K->1026125K(2084096K), >>>>>>>> 0.0072150 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 3/26/2010 10:33 AM, Siraj Haider wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hi Mike, >>>>>>>>> I am attaching the dump that I created by putting >>>>>>>>> -XX:+PrintClassHistogram in catalina options and by issuing a kill >>>>>>>>> -3 >>>>>>>>> command. The machine was not in a bad state (i.e. it was not doing >>>>>>>>> frequent Full GC cycles yet) but the kill -3 forced it to do a Full >>>>>>>>> GC. >>>>>>>>> If you want I will wait for the machine doing frequent gc cycle and >>>>>>>>> will >>>>>>>>> send you another another dump. >>>>>>>>> >>>>>>>>> thanks >>>>>>>>> -siraj >>>>>>>>> >>>>>>>>> On 3/26/2010 9:29 AM, Michael McCandless wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> I think there are various ways.... >>>>>>>>>> >>>>>>>>>> jmap would be great. >>>>>>>>>> >>>>>>>>>> Hmm looks like this may be relevant: >>>>>>>>>> >>>>>>>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6797870 >>>>>>>>>> >>>>>>>>>> I also use YourKit and watch the allocations... >>>>>>>>>> >>>>>>>>>> Mike >>>>>>>>>> >>>>>>>>>> On Thu, Mar 25, 2010 at 5:26 PM, Siraj Haider >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> how should I get that memory dump? using jmap? >>>>>>>>>>> >>>>>>>>>>> -siraj >>>>>>>>>>> >>>>>>>>>>> On 3/25/2010 4:32 PM, Michael McCandless wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Are you using IndexReader.reopen to open those new searchers? >>>>>>>>>>>> >>>>>>>>>>>> Can you capture a memory dump when GC kicks in? I'd be curious >>>>>>>>>>>> to >>>>>>>>>>>> see >>>>>>>>>>>> where all the "new" garbage is coming from... I don't think >>>>>>>>>>>> 2.3.2 >>>>>>>>>>>> -> >>>>>>>>>>>> 2.9.2 should be generating more garbage. >>>>>>>>>>>> >>>>>>>>>>>> Mike >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 25, 2010 at 3:55 PM, Siraj Haider>>>>>>>>>>> > >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Indexing happens in a different thread on intervals. I open a >>>>>>>>>>>>> new >>>>>>>>>>>>> IndexWriter for each indexing session. After indexing session >>>>>>>>>>>>> if >>>>>>>>>>>>> there >>>>>>>>>>>>> is a >>>>>>>>>>>>> modification in the index I close the searcher and open a new >>>>>>>>>>>>> one. >>>>>>>>>>>>> I >>>>>>>>>>>>> have >>>>>>>>>>>>> two searchers that I flip flop when opening an index. >>>>>>>>>>>>> >>>>>>>>>>>>> On 3/25/2010 3:26 PM, Michael McCandless wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> How do you reopen your searchers after indexing? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Do you keep a single IW open for all time? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mike >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Mar 25, 2010 at 3:11 PM, Siraj Haider< >>>>>>>>>>>>>> siraj@jobdiva.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Indexing happen with frequent intervals on our indexes, but I >>>>>>>>>>>>>>> think >>>>>>>>>>>>>>> searching is the cause of the issue, because as soon as the >>>>>>>>>>>>>>> indexes >>>>>>>>>>>>>>> are >>>>>>>>>>>>>>> hit >>>>>>>>>>>>>>> with a lot of searches, the gc cycles become more frequent. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -siraj >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 3/24/2010 5:19 PM, Michael McCandless wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is this during indexing or searching? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Mike >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Mar 24, 2010 at 3:45 PM, Grant >>>>>>>>>>>>>>>> Ingersoll >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mar 24, 2010, at 2:13 PM, Siraj Haider wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> We upgraded to 2.9.2 from 2.3.2 and the garbage collection >>>>>>>>>>>>>>>>>> performance >>>>>>>>>>>>>>>>>> deteriorated drastically. The system is going to Full GC >>>>>>>>>>>>>>>>>> cycles >>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>> long >>>>>>>>>>>>>>>>>> pauses very frequently. Did something got changed that we >>>>>>>>>>>>>>>>>> need >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> account >>>>>>>>>>>>>>>>>> for? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes, quite a bit has changed under the hood between those >>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>> versions. >>>>>>>>>>>>>>>>> CHANGES.txt is probably the best place to start. >>>>>>>>>>>>>>>>> Otherwise, >>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>> info >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> needed to figure out what is going on. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> thanks in advance >>>>>>>>>>>>>>>>>> -siraj >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org >>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -------------------------- >>>>>>>>>>>>>>>>> Grant Ingersoll >>>>>>>>>>>>>>>>> http://www.lucidimagination.com/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Search the Lucene ecosystem using Solr/Lucene: >>>>>>>>>>>>>>>>> http://www.lucidimagination.com/search >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org >>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org >>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>> java-user-help@lucene.apache.org >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org >>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>> java-user-help@lucene.apache.org >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org >>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>> java-user-help@lucene.apache.org >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >> For additional commands, e-mail: java-user-help@lucene.apache.org >> >> >> > > --00504502ad20c162160482e6fd2d--