ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 胡永亮/Bob <hu...@neusoft.com>
Subject 回复: Ignite used memory 7 times greater than imported data
Date Wed, 13 Jul 2016 07:49:02 GMT
Now, because I created my cache in java code, so changing the config.xml doesn't work.

I already change the java code of creating my cache to change the backups to zero from one:
            
CacheConfiguration<K, V> cfg =
CacheConfig.cache(cacheName, pojoStoreFactory);
//            cfg.setBackups(1);
            cfg.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
            cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
            cfg.setCacheMode(CacheMode.PARTITIONED);

After that, the same data consumed memory: 130G, the detail is the following:
[root@ignite15 apache-ignite-fabric-1.6.0-bin]# free -g
             total       used       free     shared    buffers     cached
Mem:           125         66         59          0          0          0
-/+ buffers/cache:         65         60
Swap:            3          0          3

[root@localhost apache-ignite-fabric-1.6.0-bin]# free -g
             total       used       free     shared    buffers     cached
Mem:           125         65         60          0          0          0
-/+ buffers/cache:         64         61
Swap:            3          0          3

Can anyone sugguest me how to reduce the number of used memory? 

Ignite consume too much memory!!!

Thanks.



Bob
 
发件人: 胡永亮/Bob
发送时间: 2016-07-12 17:22
收件人: user@ignite.apache.org
主题: Ignite used memory 7 times greater than imported data
Hi, everyone

    I meet a problem: Ignite used 220G+ memory, but the imported data is only 31G. Why?

    Basic info: I used Ignite 1.6, and deployed ignite cluster using 2 machine whose memory
is 128G.
    I run 3 ignite instance in every machine. I am using JDK8.

    I has 31G data in csv file. First, I import the data into oracle's one table, total records:
47,535,542
    Then I using ignite-schema-import.sh tool to produce POJO files, and put them info my
java project.
    Then I used IgniteCache.loadCache to load data from oracle to ignite. It used time: 01:43:17

    After importing data, I used linux command free and ps aux, get the following info:
    free -g:
        machine1:
                            total       used       free     shared    buffers     cached
        Mem:           125        114         11          0          0          0
        -/+ buffers/cache:        114         11
        Swap:            3          0          3
        
        machine2:
                            total       used       free     shared    buffers     cached
        Mem:           125        122          3          0          0          0
        -/+ buffers/cache:        121          4
        Swap:            3          0          3

    ps aux:
    machine1:
    USER     PID    %CPU %MEM VSZ      RSS      TTY   STAT START TIME COMMAND
    root      23234 48.2 28.8 56278844 38088436 pts/1 Sl 14:29  70:08 /usr/java/jdk1.8.0_91/bin/java
-server -Xms1g -Xmx1g -XX:NewSize=512m -XX:SurvivorRatio=6 -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=4 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M
-XX:ConcGCThreads=16 -XX:G1HeapWastePercent=10 -XX:+UseTLAB -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
-DIGNITE_QUIET=true -DIGNITE_SUCCESS_FILE=/root/apache-ignite-fabric-1.6.0-bin/work/ignite_success_6f17738f-6b73-4681-a06e-a9d270c0fe9c
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49164 -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -DIGNITE_HOME=/root/apache-ignite-fabric-1.6.0-bin
-DIGNITE_PROG_NAME=bin/ignite.sh -cp /root/apache-ignite-fabric-1.6.0-bin/libs/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-indexing/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-rest-http/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-spring/*:/root/apache-ignite-fabric-1.6.0-bin/libs/licenses/*
org.apache.ignite.startup.cmdline.CommandLineStartup config_poc.xml
root      22259 46.5 32.7 61434264 43266272 pts/1 Sl 14:28  68:14 /usr/java/jdk1.8.0_91/bin/java
...
root      22729 44.7 28.3 55533284 37374932 pts/1 Sl 14:28  65:29 /usr/java/jdk1.8.0_91/bin/java
...

    machine2:
    root     182359 76.1 32.0 60570656 42322148 pts/2 Sl 14:34  87:02 /usr/java/jdk1.8.0_91/bin/java
...
    root     181882 75.8 31.7 60219724 41962436 pts/2 Sl 14:34  87:04 /usr/java/jdk1.8.0_91/bin/java
...
    root     182867 56.8 31.8 60243104 42005888 pts/2 Sl 14:34  64:54 /usr/java/jdk1.8.0_91/bin/java
...

    The ignite config:
    <?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<!--
    Ignite Spring configuration file to startup Ignite cache.

    This file demonstrates how to configure cache using Spring. Provided cache
    will be created on node startup.

    Use this configuration file when running HTTP REST examples (see 'examples/rest' folder).

    When starting a standalone node, you need to execute the following command:
    {IGNITE_HOME}/bin/ignite.{bat|sh} examples/config/example-cache.xml

    When starting Ignite from Java IDE, pass path to this file to Ignition:
    Ignition.start("examples/config/example-cache.xml");
-->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <property name="cacheConfiguration">
            <list>
                <!-- Partitioned cache example configuration (Atomic mode). -->
                <bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="atomicityMode" value="ATOMIC"/>

<!-- Set cache mode. -->
<property name="cacheMode" value="PARTITIONED"/>

                    <property name="backups" value="1"/>

<property name="memoryMode" value="OFFHEAP_TIERED" />

<!-- Enable off-heap storage with unlimited size. -->
<!--property name="offHeapMaxMemory" value="0"/-->

<!-- Set initial cache capacity to ~ 100M. -->
<!--property name="startSize" value="#{500 * 1024 * 1024}"/--> 

<!--property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
  
  <property name="maxSize" value="100000"/>
</bean>
</property-->


                </bean>
            </list>
        </property>

<!-- Enable only some events and leave other ones disabled. -->
<!--property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
</list>
</property-->



        <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <!--
                        Ignite provides several options for automatic discovery that can be
used
                        instead os static IP based discovery. For information on all options
refer
                        to our documentation: http://apacheignite.readme.io/docs/cluster-config
                    -->
                    <!-- Uncomment static IP finder to enable static-based discovery of
initial nodes. -->
                    <!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                        <property name="addresses">
                            <list>
                                <!-- In distributed environment, replace with actual host
IP address. -->
<value>10.9.22.15:47500..47509</value>
                                <value>10.9.22.16:47500..47509</value> 
                                <value>192.168.175.39:47500..47509</value> 
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

    I don't know how analye this problem, please help me . Thanks very much. 



Bob

---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)

is intended only for the use of the intended recipient and may be confidential and/or privileged
of 
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
is 
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
copying 
is strictly prohibited, and may be unlawful.If you have received this communication in error,please

immediately notify the sender by return e-mail, and delete the original message and all copies
from 
your system. Thank you. 
---------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged
of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
is
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies
from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Mime
View raw message