ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhengqingzheng <zhengqingzh...@huawei.com>
Subject re: Ignite cache data size problem.
Date Tue, 26 Apr 2016 12:18:45 GMT
Hi Vladimir,
No problem.
I will re-run the loading process and give you  the log file.
To be clear, when you say log file, do you mean files located at work/log/*?
I use IgntieCache.loadCache(null, “select * from table”) to load all the data.

Best regards,
Kevin

发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com]
发送时间: 2016年4月26日 20:15
收件人: user@ignite.apache.org
主题: Re: Ignite cache data size problem.

Hi Kevin,

Could you please re-run your case and attach Ignite logs and GC logs from all participating
servers? I would expect that this is either a kind of out-of-memory problem, or network saturation.
Also please explain how exactly do you load data to Ignite? Do you use DataStreamer, or may
be IgniteCache.loadlCache, or IgniteCache.put?

Vladimir.

On Tue, Apr 26, 2016 at 3:01 PM, Zhengqingzheng <zhengqingzheng@huawei.com<mailto:zhengqingzheng@huawei.com>>
wrote:
Hi Vladimir,
Thank you for your help.
I tried to load 1Million records, and calculate each object’s size (including key and value
object, where key is  string type), summarized together and found the total memory consumption
is 130Mb.

Because ./ignitevisor.sh  command only shows number of records, and no data allocation information,
I don’t know how much memory has been consumed for each type of cache.

My result is as follows:
Log:
Found big object:     [Ljava.util.HashMap$Node;@24833164 size: 30.888206481933594Mb
Found big object:   java.util.HashMap@15819383<mailto:java.util.HashMap@15819383> size:
30.88824462890625Mb
Found big object: java.util.HashSet@10236420<mailto:java.util.HashSet@10236420> size:
30.888259887695312Mb
key size: 32388688 human readable data: 30.888259887695312Mb
Found big object:   [Lorg.jsr166.ConcurrentHashMap8$Node;@29556439 size: 129.99818420410156Mb
Found big object: org.jsr166.ConcurrentHashMap8@19238297<mailto:org.jsr166.ConcurrentHashMap8@19238297>
size: 129.99822235107422Mb
value size: 136313016 human readable data: 129.99822235107422Mb
the whole number of record is 47Million, so the  data size inside the cache should be 130*47=6110Mb(around
6Gb).

However, when I try to load the whole data into the cache, I still get exceptions:
The exception information is listed as follows:

1.       ---------------------------------- exception info from client ----------------------------------------
Before exception occurred, I have 10 nodes on two servers,  server1 (48G ram) has 6nodes,
each node is assigned with 7gb jvm heap; server2 (32gb ram)  has 4 nodes with the same jvm
settings as previous one.
After exception, client stopped, and 8 node left (server1’s node remain, no exception occurred
on this server), server2 ( 2 nodes remain, two nodes droped)
The total number of records loaded is 37Million.

[19:01:47] Topology snapshot [ver=77, servers=9, clients=1, CPUs=20, heap=64.0GB]
[19:01:47,463][SEVERE][pub-#46%null%][GridTaskWorker] Failed to obtain remote job result policy
for result from ComputeTask.result(..) method (will fail the whole task): GridJobResultImpl
[job=C2 [], sib=GridJobSiblingImpl [sesId=0dbc0f15451-880a2dd1-bc95-4084-a705-4effcec5d2cd,
jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, nodeId=832bed3e-dc5d-4743-9853-127e3b516924,
isJobDone=false], jobCtx=GridJobContextImpl [jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924,
timeoutObj=null, attrs={}], node=TcpDiscoveryNode [id=832bed3e-dc5d-4743-9853-127e3b516924,
addrs=[0:0:0:0:0:0:0:1%lo, 10.120.70.122, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500,
/0:0:0:0:0:0:0:1%lo:47500, /10.120.70.122:47500<http://10.120.70.122:47500>, /127.0.0.1:47500<http://127.0.0.1:47500>],
discPort=47500, order=7, intOrder=7, lastExchangeTime=1461663619161, loc=false, ver=1.5.0#20151229-sha1:f1f8cda2,
isClient=false], ex=class o.a.i.cluster.ClusterTopologyException: Node has left grid: 832bed3e-dc5d-4743-9853-127e3b516924,
hasRes=true, isCancelled=false, isOccupied=true]
class org.apache.ignite.cluster.ClusterTopologyException: Node has left grid: 832bed3e-dc5d-4743-9853-127e3b516924
         at org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1315)
         at org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1.run(GridTaskProcessor.java:1246)
         at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6453)
         at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:788)
         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)
[19:02:16] Ignite node stopped OK [uptime=01:21:52:963]

2 ------------------------------ also, I use the recommended settings for the jvm and get
the following log information ------------------------------------------------
Java HotSpot(TM) Client VM (25.66-b18) for windows-x86 JRE (1.8.0_66-b18), built on Nov  9
2015 10:58:29 by "java_re" with MS VC++ 10.0 (VS2010)
Memory: 4k page, physical 6291064k(1912832k free), swap 12580288k(6996288k free)
CommandLine flags: -XX:GCLogFileSize=104857600 -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824
-XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+UseGCLogFileRotation -XX:-UseLargePagesIndividualAllocation
2016-04-26T17:40:15.609+0800: 4.839: [Full GC (Metadata GC Threshold) 2016-04-26T17:40:15.610+0800:
4.839: [Tenured: 0K->10330K(699072K), 0.0656011 secs] 162451K->10330K(1013632K), [Metaspace:
12085K->12085K(12672K)], 0.0659450 secs] [Times: user=0.06 sys=0.00, real=0.07 secs]
2016-04-26T17:40:24.837+0800: 14.044: [Full GC (Metadata GC Threshold) 2016-04-26T17:40:24.837+0800:
14.044: [Tenured: 10330K->32179K(699072K), 0.1810716 secs] 282679K->32179K(1013632K),
[Metaspace: 20221K->20221K(20864K)], 0.1812825 secs] [Times: user=0.17 sys=0.00, real=0.18
secs]
2016-04-26T17:40:53.699+0800: 42.937: [GC (Allocation Failure) 2016-04-26T17:40:53.699+0800:
42.937: [DefNew: 279616K->8837K(314560K), 0.0242718 secs] 311795K->41016K(1013632K),
0.0244783 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-04-26T17:41:26.411+0800: 75.516: [GC (Allocation Failure) 2016-04-26T17:41:26.412+0800:
75.516: [DefNew: 288453K->2685K(314560K), 0.0116993 secs] 320632K->34864K(1013632K),
0.0119035 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:42:01.895+0800: 110.834: [GC (Allocation Failure) 2016-04-26T17:42:01.895+0800:
110.834: [DefNew: 282301K->2689K(314560K), 0.0111799 secs] 314480K->34868K(1013632K),
0.0113629 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:42:37.354+0800: 145.818: [GC (Allocation Failure) 2016-04-26T17:42:37.354+0800:
145.818: [DefNew: 282305K->2695K(314560K), 0.0102415 secs] 314484K->34874K(1013632K),
0.0104181 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:43:13.549+0800: 181.641: [GC (Allocation Failure) 2016-04-26T17:43:13.549+0800:
181.641: [DefNew: 282311K->2893K(314560K), 0.0134864 secs] 314490K->35072K(1013632K),
0.0136970 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:43:50.453+0800: 218.129: [GC (Allocation Failure) 2016-04-26T17:43:50.453+0800:
218.129: [DefNew: 282509K->2708K(314560K), 0.0160308 secs] 314688K->34887K(1013632K),
0.0162459 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-04-26T17:44:27.317+0800: 254.604: [GC (Allocation Failure) 2016-04-26T17:44:27.317+0800:
254.605: [DefNew: 282324K->2715K(314560K), 0.0128055 secs] 314503K->34894K(1013632K),
0.0130086 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:45:03.944+0800: 291.040: [GC (Allocation Failure) 2016-04-26T17:45:03.944+0800:
291.040: [DefNew: 282331K->2724K(314560K), 0.0109078 secs] 314510K->34903K(1013632K),
0.0110838 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:45:41.069+0800: 327.816: [GC (Allocation Failure) 2016-04-26T17:45:41.069+0800:
327.816: [DefNew: 282340K->2732K(314560K), 0.0128223 secs] 314519K->34912K(1013632K),
0.0130151 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:46:18.108+0800: 364.538: [GC (Allocation Failure) 2016-04-26T17:46:18.108+0800:
364.538: [DefNew: 282348K->2738K(314560K), 0.0123320 secs] 314528K->34917K(1013632K),
0.0125273 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:46:54.492+0800: 400.975: [GC (Allocation Failure) 2016-04-26T17:46:54.492+0800:
400.975: [DefNew: 282354K->2747K(314560K), 0.0126284 secs] 314533K->34926K(1013632K),
0.0128229 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:47:36.805+0800: 443.099: [GC (Allocation Failure) 2016-04-26T17:47:36.805+0800:
443.099: [DefNew: 282363K->2755K(314560K), 0.0137048 secs] 314542K->34934K(1013632K),
0.0139138 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:48:13.500+0800: 480.054: [GC (Allocation Failure) 2016-04-26T17:48:13.500+0800:
480.054: [DefNew: 282371K->2764K(314560K), 0.0134777 secs] 314550K->34943K(1013632K),
0.0137571 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:48:50.568+0800: 517.242: [GC (Allocation Failure) 2016-04-26T17:48:50.568+0800:
517.242: [DefNew: 282380K->2769K(314560K), 0.0121474 secs] 314559K->34948K(1013632K),
0.0125373 secs] [Times: user=0.00 sys=0.02, real=0.02 secs]
2016-04-26T17:49:28.506+0800: 555.103: [GC (Allocation Failure) 2016-04-26T17:49:28.507+0800:
555.103: [DefNew: 282385K->2777K(314560K), 0.0133794 secs] 314564K->34957K(1013632K),
0.0135732 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:50:05.869+0800: 592.355: [GC (Allocation Failure) 2016-04-26T17:50:05.869+0800:
592.355: [DefNew: 282393K->1713K(314560K), 0.0129486 secs] 314573K->34965K(1013632K),
0.0131366 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:50:43.336+0800: 629.663: [GC (Allocation Failure) 2016-04-26T17:50:43.336+0800:
629.663: [DefNew: 281329K->1688K(314560K), 0.0107519 secs] 314581K->34971K(1013632K),
0.0109511 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:51:20.603+0800: 666.805: [GC (Allocation Failure) 2016-04-26T17:51:20.604+0800:
666.806: [DefNew: 281304K->2043K(314560K), 0.0125329 secs] 314587K->35333K(1013632K),
0.0127661 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:51:57.981+0800: 704.195: [GC (Allocation Failure) 2016-04-26T17:51:57.981+0800:
704.195: [DefNew: 281659K->1688K(314560K), 0.0083589 secs] 314949K->34985K(1013632K),
0.0085578 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:52:35.234+0800: 741.413: [GC (Allocation Failure) 2016-04-26T17:52:35.234+0800:
741.413: [DefNew: 281304K->1686K(314560K), 0.0080466 secs] 314601K->34990K(1013632K),
0.0082145 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:53:11.734+0800: 778.046: [GC (Allocation Failure) 2016-04-26T17:53:11.734+0800:
778.046: [DefNew: 281302K->1689K(314560K), 0.0106016 secs] 314606K->35000K(1013632K),
0.0108028 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:53:48.679+0800: 814.992: [GC (Allocation Failure) 2016-04-26T17:53:48.679+0800:
814.992: [DefNew: 281305K->1689K(314560K), 0.0085276 secs] 314616K->35007K(1013632K),
0.0086721 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:54:24.734+0800: 851.078: [GC (Allocation Failure) 2016-04-26T17:54:24.735+0800:
851.078: [DefNew: 281305K->1685K(314560K), 0.0130525 secs] 314623K->35012K(1013632K),
0.0132933 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:55:01.614+0800: 888.090: [GC (Allocation Failure) 2016-04-26T17:55:01.614+0800:
888.090: [DefNew: 281301K->1686K(314560K), 0.0084343 secs] 314628K->35021K(1013632K),
0.0086352 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:55:38.728+0800: 925.129: [GC (Allocation Failure) 2016-04-26T17:55:38.728+0800:
925.129: [DefNew: 281302K->1684K(314560K), 0.0081086 secs] 314637K->35026K(1013632K),
0.0082835 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:56:15.257+0800: 961.859: [GC (Allocation Failure) 2016-04-26T17:56:15.257+0800:
961.859: [DefNew: 281300K->1683K(314560K), 0.0097599 secs] 314642K->35032K(1013632K),
0.0099144 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:56:51.417+0800: 998.123: [GC (Allocation Failure) 2016-04-26T17:56:51.417+0800:
998.123: [DefNew: 281299K->1681K(314560K), 0.0095046 secs] 314648K->35038K(1013632K),
0.0096970 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:57:27.838+0800: 1034.605: [GC (Allocation Failure) 2016-04-26T17:57:27.838+0800:
1034.605: [DefNew: 281297K->1681K(314560K), 0.0104843 secs] 314654K->35045K(1013632K),
0.0106871 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:58:04.008+0800: 1070.815: [GC (Allocation Failure) 2016-04-26T17:58:04.008+0800:
1070.815: [DefNew: 281297K->1680K(314560K), 0.0087810 secs] 314661K->35051K(1013632K),
0.0089486 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:58:40.916+0800: 1107.629: [GC (Allocation Failure) 2016-04-26T17:58:40.916+0800:
1107.629: [DefNew: 281296K->1679K(314560K), 0.0123532 secs] 314667K->35058K(1013632K),
0.0125745 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:59:17.297+0800: 1143.843: [GC (Allocation Failure) 2016-04-26T17:59:17.297+0800:
1143.843: [DefNew: 281295K->1682K(314560K), 0.0099356 secs] 314674K->35069K(1013632K),
0.0101206 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:59:54.265+0800: 1180.679: [GC (Allocation Failure) 2016-04-26T17:59:54.265+0800:
1180.680: [DefNew: 281298K->1678K(314560K), 0.0089453 secs] 314685K->35071K(1013632K),
0.0091503 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:00:31.910+0800: 1218.043: [GC (Allocation Failure) 2016-04-26T18:00:31.910+0800:
1218.043: [DefNew: 281294K->1677K(314560K), 0.0082765 secs] 314687K->35078K(1013632K),
0.0084452 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:01:07.853+0800: 1254.048: [GC (Allocation Failure) 2016-04-26T18:01:07.854+0800:
1254.048: [DefNew: 281293K->1678K(314560K), 0.0089447 secs] 314694K->35085K(1013632K),
0.0091405 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:01:44.497+0800: 1290.474: [GC (Allocation Failure) 2016-04-26T18:01:44.497+0800:
1290.474: [DefNew: 281294K->1678K(314560K), 0.0076934 secs] 314701K->35091K(1013632K),
0.0078666 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:02:27.863+0800: 1333.673: [GC (Allocation Failure) 2016-04-26T18:02:27.863+0800:
1333.674: [DefNew: 281294K->1674K(314560K), 0.0084192 secs] 314707K->35098K(1013632K),
0.0085597 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:03:04.588+0800: 1370.424: [GC (Allocation Failure) 2016-04-26T18:03:04.589+0800:
1370.424: [DefNew: 281290K->1677K(314560K), 0.0086385 secs] 314714K->35107K(1013632K),
0.0087911 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:03:41.646+0800: 1407.396: [GC (Allocation Failure) 2016-04-26T18:03:41.646+0800:
1407.396: [DefNew: 281293K->1674K(314560K), 0.0079049 secs] 314723K->35111K(1013632K),
0.0080862 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:04:18.618+0800: 1444.458: [GC (Allocation Failure) 2016-04-26T18:04:18.618+0800:
1444.458: [DefNew: 281290K->1674K(314560K), 0.0089752 secs] 314727K->35118K(1013632K),
0.0091841 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:04:55.359+0800: 1481.281: [GC (Allocation Failure) 2016-04-26T18:04:55.359+0800:
1481.281: [DefNew: 281290K->1674K(314560K), 0.0091816 secs] 314734K->35124K(1013632K),
0.0093663 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:05:31.696+0800: 1517.690: [GC (Allocation Failure) 2016-04-26T18:05:31.696+0800:
1517.690: [DefNew: 281290K->1680K(314560K), 0.0097599 secs] 314740K->35136K(1013632K),
0.0099613 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:06:08.293+0800: 1554.357: [GC (Allocation Failure) 2016-04-26T18:06:08.293+0800:
1554.357: [DefNew: 281296K->1681K(314560K), 0.0102206 secs] 314752K->35144K(1013632K),
0.0104262 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:06:44.886+0800: 1590.882: [GC (Allocation Failure) 2016-04-26T18:06:44.886+0800:
1590.882: [DefNew: 281297K->1680K(314560K), 0.0090525 secs] 314760K->35150K(1013632K),
0.0092456 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:07:20.852+0800: 1626.984: [GC (Allocation Failure) 2016-04-26T18:07:20.852+0800:
1626.984: [DefNew: 281296K->1682K(314560K), 0.0095817 secs] 314766K->35158K(1013632K),
0.0097445 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:07:57.290+0800: 1663.543: [GC (Allocation Failure) 2016-04-26T18:07:57.290+0800:
1663.543: [DefNew: 281298K->1682K(314560K), 0.0076180 secs] 314774K->35164K(1013632K),
0.0077856 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:08:34.599+0800: 1700.805: [GC (Allocation Failure) 2016-04-26T18:08:34.599+0800:
1700.805: [DefNew: 281298K->1680K(314560K), 0.0136244 secs] 314780K->35170K(1013632K),
0.0138406 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:09:10.999+0800: 1737.438: [GC (Allocation Failure) 2016-04-26T18:09:10.999+0800:
1737.438: [DefNew: 281296K->1681K(314560K), 0.0086762 secs] 314786K->35177K(1013632K),
0.0088701 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:09:47.695+0800: 1774.086: [GC (Allocation Failure) 2016-04-26T18:09:47.695+0800:
1774.086: [DefNew: 281297K->1679K(314560K), 0.0079063 secs] 314793K->35182K(1013632K),
0.0080798 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:10:24.630+0800: 1811.127: [GC (Allocation Failure) 2016-04-26T18:10:24.630+0800:
1811.127: [DefNew: 281295K->1680K(314560K), 0.0100454 secs] 314798K->35189K(1013632K),
0.0101801 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:11:00.972+0800: 1847.581: [GC (Allocation Failure) 2016-04-26T18:11:00.972+0800:
1847.581: [DefNew: 281296K->1679K(314560K), 0.0083639 secs] 314805K->35195K(1013632K),
0.0085452 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:11:37.280+0800: 1883.810: [GC (Allocation Failure) 2016-04-26T18:11:37.280+0800:
1883.810: [DefNew: 281295K->1679K(314560K), 0.0084648 secs] 314811K->35202K(1013632K),
0.0086475 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:12:13.694+0800: 1920.386: [GC (Allocation Failure) 2016-04-26T18:12:13.694+0800:
1920.386: [DefNew: 281295K->1677K(314560K), 0.0080820 secs] 314818K->35207K(1013632K),
0.0082119 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:12:49.894+0800: 1956.802: [GC (Allocation Failure) 2016-04-26T18:12:49.894+0800:
1956.803: [DefNew: 281293K->1677K(314560K), 0.0146823 secs] 314823K->35213K(1013632K),
0.0149620 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
2016-04-26T18:13:26.430+0800: 1993.317: [GC (Allocation Failure) 2016-04-26T18:13:26.430+0800:
1993.317: [DefNew: 281293K->1677K(314560K), 0.0088413 secs] 314829K->35220K(1013632K),
0.0090009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:14:03.233+0800: 2030.321: [GC (Allocation Failure) 2016-04-26T18:14:03.233+0800:
2030.321: [DefNew: 281293K->1676K(314560K), 0.0078476 secs] 314836K->35225K(1013632K),
0.0079887 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:14:40.063+0800: 2067.102: [GC (Allocation Failure) 2016-04-26T18:14:40.063+0800:
2067.102: [DefNew: 281292K->1677K(314560K), 0.0092786 secs] 314841K->35232K(1013632K),
0.0094601 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:15:16.165+0800: 2103.376: [GC (Allocation Failure) 2016-04-26T18:15:16.165+0800:
2103.376: [DefNew: 281293K->2106K(314560K), 0.0135981 secs] 314848K->35675K(1013632K),
0.0138093 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:15:52.922+0800: 2139.884: [GC (Allocation Failure) 2016-04-26T18:15:52.922+0800:
2139.884: [DefNew: 281722K->1669K(314560K), 0.0123189 secs] 315291K->35244K(1013632K),
0.0125281 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:16:30.496+0800: 2177.184: [GC (Allocation Failure) 2016-04-26T18:16:30.496+0800:
2177.184: [DefNew: 281285K->1669K(314560K), 0.0089190 secs] 314860K->35251K(1013632K),
0.0090788 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:17:06.767+0800: 2213.365: [GC (Allocation Failure) 2016-04-26T18:17:06.767+0800:
2213.365: [DefNew: 281285K->1826K(314560K), 0.0079870 secs] 314867K->35415K(1013632K),
0.0081401 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:17:43.991+0800: 2250.422: [GC (Allocation Failure) 2016-04-26T18:17:43.991+0800:
2250.422: [DefNew: 281442K->1669K(314560K), 0.0115439 secs] 315031K->35264K(1013632K),
0.0117428 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:18:20.528+0800: 2286.733: [GC (Allocation Failure) 2016-04-26T18:18:20.528+0800:
2286.733: [DefNew: 281285K->1668K(314560K), 0.0082332 secs] 314880K->35270K(1013632K),
0.0084053 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:18:57.215+0800: 2323.271: [GC (Allocation Failure) 2016-04-26T18:18:57.215+0800:
2323.271: [DefNew: 281284K->1671K(314560K), 0.0077934 secs] 314886K->35277K(1013632K),
0.0079697 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:19:34.151+0800: 2359.977: [GC (Allocation Failure) 2016-04-26T18:19:34.151+0800:
2359.977: [DefNew: 281287K->1669K(314560K), 0.0078809 secs] 314893K->35283K(1013632K),
0.0080410 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:20:11.106+0800: 2396.706: [GC (Allocation Failure) 2016-04-26T18:20:11.106+0800:
2396.706: [DefNew: 281285K->1673K(314560K), 0.0092928 secs] 314899K->35293K(1013632K),
0.0094864 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:20:47.129+0800: 2432.801: [GC (Allocation Failure) 2016-04-26T18:20:47.129+0800:
2432.801: [DefNew: 281289K->1672K(314560K), 0.0089699 secs] 314909K->35298K(1013632K),
0.0091397 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:21:23.028+0800: 2468.633: [GC (Allocation Failure) 2016-04-26T18:21:23.028+0800:
2468.633: [DefNew: 281288K->1671K(314560K), 0.0124404 secs] 314914K->35303K(1013632K),
0.0128810 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:21:59.745+0800: 2505.506: [GC (Allocation Failure) 2016-04-26T18:21:59.745+0800:
2505.506: [DefNew: 281287K->1671K(314560K), 0.0080019 secs] 314919K->35309K(1013632K),
0.0082167 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:22:36.639+0800: 2542.570: [GC (Allocation Failure) 2016-04-26T18:22:36.639+0800:
2542.570: [DefNew: 281287K->1670K(314560K), 0.0082147 secs] 314925K->35315K(1013632K),
0.0083938 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:23:12.390+0800: 2578.421: [GC (Allocation Failure) 2016-04-26T18:23:12.390+0800:
2578.421: [DefNew: 281286K->2207K(314560K), 0.0114121 secs] 314931K->35857K(1013632K),
0.0116115 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:23:48.693+0800: 2615.071: [GC (Allocation Failure) 2016-04-26T18:23:48.693+0800:
2615.071: [DefNew: 281823K->1798K(314560K), 0.0104047 secs] 315473K->35455K(1013632K),
0.0105935 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:24:25.698+0800: 2652.227: [GC (Allocation Failure) 2016-04-26T18:24:25.698+0800:
2652.228: [DefNew: 281414K->1795K(314560K), 0.0102968 secs] 315071K->35459K(1013632K),
0.0104854 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:25:01.801+0800: 2688.457: [GC (Allocation Failure) 2016-04-26T18:25:01.801+0800:
2688.458: [DefNew: 281411K->2126K(314560K), 0.0090964 secs] 315075K->35795K(1013632K),
0.0093529 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
2016-04-26T18:25:39.055+0800: 2725.732: [GC (Allocation Failure) 2016-04-26T18:25:39.055+0800:
2725.732: [DefNew: 281742K->1799K(314560K), 0.0085785 secs] 315411K->35474K(1013632K),
0.0087168 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:26:27.232+0800: 2773.853: [GC (Allocation Failure) 2016-04-26T18:26:27.232+0800:
2773.853: [DefNew: 281415K->1800K(314560K), 0.0091224 secs] 315090K->35482K(1013632K),
0.0094059 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:27:05.583+0800: 2812.453: [GC (Allocation Failure) 2016-04-26T18:27:05.584+0800:
2812.453: [DefNew: 281416K->1801K(314560K), 0.0083206 secs] 315098K->35489K(1013632K),
0.0084969 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:27:43.412+0800: 2850.382: [GC (Allocation Failure) 2016-04-26T18:27:43.412+0800:
2850.382: [DefNew: 281417K->1801K(314560K), 0.0095576 secs] 315105K->35495K(1013632K),
0.0097616 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:28:20.067+0800: 2887.191: [GC (Allocation Failure) 2016-04-26T18:28:20.067+0800:
2887.191: [DefNew: 281417K->1799K(314560K), 0.0121244 secs] 315111K->35501K(1013632K),
0.0123463 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:28:55.431+0800: 2923.064: [GC (Allocation Failure) 2016-04-26T18:28:55.431+0800:
2923.064: [DefNew: 281415K->1798K(314560K), 0.0088000 secs] 315117K->35506K(1013632K),
0.0089771 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:29:30.761+0800: 2958.335: [GC (Allocation Failure) 2016-04-26T18:29:30.761+0800:
2958.335: [DefNew: 281414K->1798K(314560K), 0.0113492 secs] 315122K->35512K(1013632K),
0.0115526 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:30:05.795+0800: 2993.528: [GC (Allocation Failure) 2016-04-26T18:30:05.795+0800:
2993.528: [DefNew: 281414K->1797K(314560K), 0.0112305 secs] 315128K->35518K(1013632K),
0.0114227 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:30:41.648+0800: 3029.322: [GC (Allocation Failure) 2016-04-26T18:30:41.648+0800:
3029.322: [DefNew: 281413K->1797K(314560K), 0.0174930 secs] 315134K->35524K(1013632K),
0.0177282 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:31:34.467+0800: 3082.237: [GC (Allocation Failure) 2016-04-26T18:31:34.467+0800:
3082.237: [DefNew: 281413K->1800K(314560K), 0.0095434 secs] 315140K->35534K(1013632K),
0.0097186 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:32:14.021+0800: 3121.810: [GC (Allocation Failure) 2016-04-26T18:32:14.021+0800:
3121.810: [DefNew: 281416K->1800K(314560K), 0.0096049 secs] 315150K->35540K(1013632K),
0.0097856 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:33:07.874+0800: 3175.537: [GC (Allocation Failure) 2016-04-26T18:33:07.874+0800:
3175.537: [DefNew: 281416K->1697K(314560K), 0.0135850 secs] 315156K->35550K(1013632K),
0.0138194 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:33:47.621+0800: 3215.029: [GC (Allocation Failure) 2016-04-26T18:33:47.621+0800:
3215.029: [DefNew: 281313K->1694K(314560K), 0.0080256 secs] 315166K->35566K(1013632K),
0.0081854 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:34:24.160+0800: 3251.370: [GC (Allocation Failure) 2016-04-26T18:34:24.160+0800:
3251.371: [DefNew: 281310K->1691K(314560K), 0.0086257 secs] 315182K->35570K(1013632K),
0.0087810 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:34:59.507+0800: 3286.713: [GC (Allocation Failure) 2016-04-26T18:34:59.507+0800:
3286.713: [DefNew: 281307K->1690K(314560K), 0.0082689 secs] 315186K->35576K(1013632K),
0.0084583 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:35:34.974+0800: 3322.008: [GC (Allocation Failure) 2016-04-26T18:35:34.974+0800:
3322.008: [DefNew: 281306K->1690K(314560K), 0.0134400 secs] 315192K->35582K(1013632K),
0.0136509 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:36:11.007+0800: 3357.907: [GC (Allocation Failure) 2016-04-26T18:36:11.007+0800:
3357.907: [DefNew: 281306K->1689K(314560K), 0.0118568 secs] 315198K->35590K(1013632K),
0.0120744 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:36:48.227+0800: 3394.956: [GC (Allocation Failure) 2016-04-26T18:36:48.227+0800:
3394.956: [DefNew: 281305K->1688K(314560K), 0.0083156 secs] 315206K->35596K(1013632K),
0.0084603 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2016-04-26T18:37:24.551+0800: 3430.991: [GC (Allocation Failure) 2016-04-26T18:37:24.551+0800:
3430.991: [DefNew: 281304K->1688K(314560K), 0.0091559 secs] 315212K->35602K(1013632K),
0.0093369 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:38:01.307+0800: 3467.736: [GC (Allocation Failure) 2016-04-26T18:38:01.307+0800:
3467.736: [DefNew: 281304K->1685K(314560K), 0.0085799 secs] 315218K->35609K(1013632K),
0.0087592 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:38:37.590+0800: 3503.935: [GC (Allocation Failure) 2016-04-26T18:38:37.590+0800:
3503.935: [DefNew: 281301K->1685K(314560K), 0.0103468 secs] 315225K->35616K(1013632K),
0.0105709 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:39:14.026+0800: 3540.397: [GC (Allocation Failure) 2016-04-26T18:39:14.026+0800:
3540.397: [DefNew: 281301K->1685K(314560K), 0.0119247 secs] 315232K->35622K(1013632K),
0.0121515 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:39:50.866+0800: 3577.377: [GC (Allocation Failure) 2016-04-26T18:39:50.866+0800:
3577.377: [DefNew: 281301K->1687K(314560K), 0.0134316 secs] 315238K->35630K(1013632K),
0.0136453 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:40:28.204+0800: 3614.598: [GC (Allocation Failure) 2016-04-26T18:40:28.204+0800:
3614.598: [DefNew: 281303K->1687K(314560K), 0.0093311 secs] 315246K->35635K(1013632K),
0.0095685 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:41:04.400+0800: 3650.792: [GC (Allocation Failure) 2016-04-26T18:41:04.400+0800:
3650.792: [DefNew: 281303K->1684K(314560K), 0.0114953 secs] 315251K->35642K(1013632K),
0.0116892 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:41:41.313+0800: 3687.569: [GC (Allocation Failure) 2016-04-26T18:41:41.313+0800:
3687.569: [DefNew: 281300K->1683K(314560K), 0.0078890 secs] 315258K->35648K(1013632K),
0.0080298 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:42:16.550+0800: 3722.774: [GC (Allocation Failure) 2016-04-26T18:42:16.550+0800:
3722.774: [DefNew: 281299K->1679K(314560K), 0.0124507 secs] 315264K->35654K(1013632K),
0.0126969 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:42:52.776+0800: 3759.034: [GC (Allocation Failure) 2016-04-26T18:42:52.776+0800:
3759.034: [DefNew: 281295K->1673K(314560K), 0.0105041 secs] 315270K->35661K(1013632K),
0.0106768 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:43:28.412+0800: 3794.638: [GC (Allocation Failure) 2016-04-26T18:43:28.412+0800:
3794.638: [DefNew: 281289K->1674K(314560K), 0.0103795 secs] 315277K->35669K(1013632K),
0.0105525 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:44:04.536+0800: 3830.831: [GC (Allocation Failure) 2016-04-26T18:44:04.536+0800:
3830.831: [DefNew: 281290K->1674K(314560K), 0.0079088 secs] 315285K->35675K(1013632K),
0.0080555 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:44:40.190+0800: 3866.449: [GC (Allocation Failure) 2016-04-26T18:44:40.190+0800:
3866.449: [DefNew: 281290K->1673K(314560K), 0.0095096 secs] 315291K->35681K(1013632K),
0.0096861 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:45:16.528+0800: 3902.765: [GC (Allocation Failure) 2016-04-26T18:45:16.528+0800:
3902.765: [DefNew: 281289K->1673K(314560K), 0.0130461 secs] 315297K->35687K(1013632K),
0.0132673 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:45:52.574+0800: 3939.001: [GC (Allocation Failure) 2016-04-26T18:45:52.574+0800:
3939.001: [DefNew: 281289K->1680K(314560K), 0.0101569 secs] 315303K->35700K(1013632K),
0.0103066 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:46:28.413+0800: 3975.223: [GC (Allocation Failure) 2016-04-26T18:46:28.413+0800:
3975.223: [DefNew: 281296K->1679K(314560K), 0.0080063 secs] 315316K->35706K(1013632K),
0.0081820 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:47:04.190+0800: 4011.234: [GC (Allocation Failure) 2016-04-26T18:47:04.190+0800:
4011.234: [DefNew: 281295K->1678K(314560K), 0.0102572 secs] 315322K->35712K(1013632K),
0.0104334 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:47:40.161+0800: 4047.120: [GC (Allocation Failure) 2016-04-26T18:47:40.161+0800:
4047.120: [DefNew: 281294K->1677K(314560K), 0.0109581 secs] 315328K->35718K(1013632K),
0.0111609 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:48:15.352+0800: 4082.428: [GC (Allocation Failure) 2016-04-26T18:48:15.352+0800:
4082.428: [DefNew: 281293K->1677K(314560K), 0.0087567 secs] 315334K->35724K(1013632K),
0.0089472 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:48:51.254+0800: 4118.407: [GC (Allocation Failure) 2016-04-26T18:48:51.254+0800:
4118.407: [DefNew: 281293K->1677K(314560K), 0.0115241 secs] 315340K->35731K(1013632K),
0.0117504 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:49:26.547+0800: 4153.705: [GC (Allocation Failure) 2016-04-26T18:49:26.547+0800:
4153.705: [DefNew: 281293K->1967K(314560K), 0.0095867 secs] 315347K->36027K(1013632K),
0.0097683 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:50:02.043+0800: 4189.206: [GC (Allocation Failure) 2016-04-26T18:50:02.043+0800:
4189.206: [DefNew: 281583K->1676K(314560K), 0.0096753 secs] 315643K->35742K(1013632K),
0.0098599 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:50:37.663+0800: 4224.505: [GC (Allocation Failure) 2016-04-26T18:50:37.663+0800:
4224.505: [DefNew: 281292K->1675K(314560K), 0.0077731 secs] 315358K->35749K(1013632K),
0.0079524 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:51:13.058+0800: 4259.780: [GC (Allocation Failure) 2016-04-26T18:51:13.058+0800:
4259.780: [DefNew: 281291K->1676K(314560K), 0.0086972 secs] 315365K->35756K(1013632K),
0.0088771 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:51:48.677+0800: 4295.269: [GC (Allocation Failure) 2016-04-26T18:51:48.677+0800:
4295.269: [DefNew: 281292K->1931K(314560K), 0.0099264 secs] 315372K->36016K(1013632K),
0.0101337 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:52:23.956+0800: 4330.493: [GC (Allocation Failure) 2016-04-26T18:52:23.956+0800:
4330.493: [DefNew: 281547K->1673K(314560K), 0.0079896 secs] 315632K->35766K(1013632K),
0.0081558 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:52:59.459+0800: 4366.025: [GC (Allocation Failure) 2016-04-26T18:52:59.459+0800:
4366.025: [DefNew: 281289K->1942K(314560K), 0.0099881 secs] 315382K->36041K(1013632K),
0.0101999 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:53:35.248+0800: 4401.571: [GC (Allocation Failure) 2016-04-26T18:53:35.248+0800:
4401.571: [DefNew: 281558K->1675K(314560K), 0.0080323 secs] 315657K->35779K(1013632K),
0.0081949 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:54:11.231+0800: 4437.434: [GC (Allocation Failure) 2016-04-26T18:54:11.231+0800:
4437.434: [DefNew: 281291K->1672K(314560K), 0.0107709 secs] 315395K->35785K(1013632K),
0.0109682 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:54:47.360+0800: 4473.741: [GC (Allocation Failure) 2016-04-26T18:54:47.360+0800:
4473.741: [DefNew: 281288K->1675K(314560K), 0.0077845 secs] 315401K->35796K(1013632K),
0.0079611 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:55:22.937+0800: 4509.361: [GC (Allocation Failure) 2016-04-26T18:55:22.937+0800:
4509.361: [DefNew: 281291K->1675K(314560K), 0.0082094 secs] 315412K->35802K(1013632K),
0.0083421 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:55:58.273+0800: 4544.708: [GC (Allocation Failure) 2016-04-26T18:55:58.273+0800:
4544.708: [DefNew: 281291K->1674K(314560K), 0.0100806 secs] 315418K->35807K(1013632K),
0.0102764 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:56:33.184+0800: 4579.506: [GC (Allocation Failure) 2016-04-26T18:56:33.184+0800:
4579.507: [DefNew: 281290K->1674K(314560K), 0.0111419 secs] 315423K->35814K(1013632K),
0.0113590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:57:09.091+0800: 4615.482: [GC (Allocation Failure) 2016-04-26T18:57:09.091+0800:
4615.482: [DefNew: 281290K->1672K(314560K), 0.0139571 secs] 315430K->35818K(1013632K),
0.0141755 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:57:45.582+0800: 4651.989: [GC (Allocation Failure) 2016-04-26T18:57:45.582+0800:
4651.989: [DefNew: 281288K->1672K(314560K), 0.0089576 secs] 315434K->35824K(1013632K),
0.0091738 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:58:21.593+0800: 4688.001: [GC (Allocation Failure) 2016-04-26T18:58:21.593+0800:
4688.002: [DefNew: 281288K->1676K(314560K), 0.0158752 secs] 315440K->35835K(1013632K),
0.0160361 secs] [Times: user=0.00 sys=0.00, real=0.02 secs]
2016-04-26T18:58:56.204+0800: 4722.662: [GC (Allocation Failure) 2016-04-26T18:58:56.204+0800:
4722.662: [DefNew: 281292K->1675K(314560K), 0.0076781 secs] 315451K->35841K(1013632K),
0.0078622 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:59:31.386+0800: 4757.760: [GC (Allocation Failure) 2016-04-26T18:59:31.386+0800:
4757.760: [DefNew: 281291K->1678K(314560K), 0.0079359 secs] 315457K->35849K(1013632K),
0.0080991 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:00:07.775+0800: 4794.332: [GC (Allocation Failure) 2016-04-26T19:00:07.775+0800:
4794.332: [DefNew: 281294K->1676K(314560K), 0.0087637 secs] 315465K->35854K(1013632K),
0.0089416 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T19:00:44.243+0800: 4830.820: [GC (Allocation Failure) 2016-04-26T19:00:44.243+0800:
4830.820: [DefNew: 281292K->1674K(314560K), 0.0098652 secs] 315470K->35859K(1013632K),
0.0101359 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:01:19.025+0800: 4865.773: [GC (Allocation Failure) 2016-04-26T19:01:19.025+0800:
4865.773: [DefNew: 281290K->1686K(314560K), 0.0077619 secs] 315475K->35877K(1013632K),
0.0079340 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:01:47.622+0800: 4894.541: [GC (Allocation Failure) 2016-04-26T19:01:47.622+0800:
4894.541: [DefNew: 281302K->2540K(314560K), 0.0189695 secs] 315493K->36737K(1013632K),
0.0192055 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
Heap
def new generation   total 314560K, used 268425K [0x04600000, 0x19b50000, 0x19b50000)
  eden space 279616K,  95% used [0x04600000, 0x149a73e0, 0x15710000)
  from space 34944K,   7% used [0x17930000, 0x17bab258, 0x19b50000)
  to   space 34944K,   0% used [0x15710000, 0x15710000, 0x17930000)
tenured generation   total 699072K, used 34196K [0x19b50000, 0x44600000, 0x44600000)
   the space 699072K,   4% used [0x19b50000, 0x1bcb5300, 0x1bcb5400, 0x44600000)
Metaspace       used 21496K, capacity 21761K, committed 21888K, reserved 21888K


发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com<mailto:vozerov@gridgain.com>]
发送时间: 2016年4月25日 20:21
收件人: user@ignite.apache.org<mailto:user@ignite.apache.org>
主题: Re: Ignite cache data size problem.

Hi Kevin,

I performed several experiments. Essentially, I put 1M entries of the class you provided with
fields initialized as follows:


for (int i = 0; i < 1_000_000; i++) {
    UniqueField field = new UniqueField();

    field.setDate(new Date());
    field.setGuid(UUID.randomUUID().toString());
    field.setMsg(String.valueOf(i));
    field.setNum(BigDecimal.valueOf(ThreadLocalRandom.current().nextDouble()));
    field.setOId(String.valueOf(i));
    field.setOrgId(String.valueOf(i));

    cache.put(i, field);
}

My results are:
1) Onheap, no indexes - about 400Mb is required to store 1M objects, or ~20Gb for 47M objects.
2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
3) Offheap, with indexes - about 400Mb offheap memory is required, or ~20Gb for all your objects.

Could you please provide more information on the error you receive? Also, you could try load
entries in a batches of a well-defined size (say, 1M), and see what happens to the system.
I expect you should see similar numbers.

Vladimir.

On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zhengqingzheng@huawei.com<mailto:zhengqingzheng@huawei.com>>
wrote:
BTW, I created 4 + 3 nodes on two servers.
each node I called command like this ./ignite.sh  -J-Xmx8g -J-Xms8g

kind regards,
Kevin



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Mime
View raw message