ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elmers, Mario (M)" <MElm...@dow.com>
Subject RE: Question Ignite Memory Consumption / Object size (Apache Ignite .NET)
Date Fri, 20 Oct 2017 07:35:56 GMT
Hi Pavel,

After doing some research, with QuerySQLField and Indexes, also with the heap size,
it seems that with each QuerySQLField the amount of memory needed is doubled by the size of
the
field.

All test are done by 3 Windows 2012R2 Nodes with Apache.Ignite 2.2 heap size 1GB per java
switch
7.5 GB Data ASCII Textfiles
64 million rows of data
Key is a guid data type

So my assumption was that I need a little more than 21GB
7.5 GB of ASCII data        ->                            15GB of UTF data
64 million keys of Guid  ->                               1 GB
Cache overhead 3 * 300MB  ->                   1.2 GB
Static assigned Heap 1GB per Node ->       3 GB
                                                                --------------------------------
                                                                Sum     20.2 GB

test 1
data load with No QueryEntities defined. -> 21.4 GB used memory

test 2
data load with QueryEntities defined but no [QuerySQLField] attributes. -> 22.2 GB used
memory

test 3
data load with QueryEntities defined and [QuerySQLField] attributes. -> 38,7 GB used memory

After running test 3 it seems that when I use QuerySQLField then the memory needed is doubled
by my estimation. The difference between test 2 and test 3 is 16.5 GB of memory needed.

src:

DataItem this is the only class which is used. It’s a simple class for only getting the
data into the db.


    public class DataItem : IBinarizable
    {
        [QuerySqlField(IsIndexed = true)]
        public DateTime DateTime;
        [QuerySqlField]
        public short FracSec;
        [QuerySqlField(IsIndexed = true)]
        public string EventType = "";
        [QuerySqlField(IsIndexed = true)]
        public string Category = "";
        [QuerySqlField(IsIndexed = true)]
        public string Area = "";
        [QuerySqlField]
        public string Node = "";
        [QuerySqlField]
        public string Unit = "";
        [QuerySqlField(IsIndexed = true)]
        public string Module = "";
        [QuerySqlField]
        public string Module_Description = "";
        [QuerySqlField]
        public string Attribute = "";
        [QuerySqlField(IsIndexed = true)]
        public string State = "";
        [QuerySqlField(IsIndexed = true)]
        public string Level = "";
        [QuerySqlField]
        public string Desc1 = "";
        [QuerySqlField]
        public string Desc2 = "";
        [QuerySqlField]
        public string Desc3 = "";
        [QuerySqlField]
        public string Desc4 = "";


        public void ReadBinary(IBinaryReader reader)
        {
            Area = reader.ReadString("Area");
            Attribute = reader.ReadString("Attribute");
            Category =reader.ReadString("Category");
            DateTime? tmp =             reader.ReadTimestamp("DateTime");
            if (tmp.HasValue)
                DateTime = tmp.Value;
            Desc1 =reader.ReadString("Desc1");
            Desc2=reader.ReadString("Desc2");
            Desc3 =reader.ReadString("Desc3");
            Desc4=reader.ReadString("Desc4");
            EventType =reader.ReadString("EventType" );
            FracSec= reader.ReadShort("FracSec");
            Level=reader.ReadString("Level");
            Module=reader.ReadString("Module");
            Module_Description=reader.ReadString("Module_Description");
            Node=reader.ReadString("Node");
            State=reader.ReadString("State");
            Unit=reader.ReadString("Unit");
        }

        public void WriteBinary(IBinaryWriter writer)
        {
            writer.WriteString("Area", Area);
            writer.WriteString("Attribute", Attribute);
            writer.WriteString("Category", Category);
            writer.WriteTimestamp("DateTime", DateTime);
            writer.WriteString("Desc1", Desc1);
            writer.WriteString("Desc2", Desc2);
            writer.WriteString("Desc3", Desc3);
            writer.WriteString("Desc4", Desc4);
            writer.WriteString("EventType", EventType);
            writer.WriteShort("FracSec", FracSec);
            writer.WriteString("Level", Level);
            writer.WriteString("Module", Module);
            writer.WriteString("Module_Description", Module_Description);
            writer.WriteString("Node", Node);
            writer.WriteString("State", State);
            writer.WriteString("Unit", Unit);
        }
    }

Apache.Ignite.exe.config addition:
<igniteConfiguration xmlns="http://ignite.apache.org/schema/dotnet/IgniteConfigurationSection">
    <!-- Customize Ignite configuration here. -->
    <memoryConfiguration defaultMemoryPolicyName="meinTest">
      <memoryPolicies>
        <memoryPolicyConfiguration name="meinTest" initialSize="5368709120" maxSize="10737418240"/>
      </memoryPolicies>
    </memoryConfiguration>
  </igniteConfiguration>


DataLoader code cache configuration:
CacheConfiguration myAECache = new CacheConfiguration() ;
myAECache.CacheMode = CacheMode.Partitioned;
myAECache.WriteThrough = false;
myAECache.ReadThrough = false;
myAECache.Backups = 0;
myAECache.Name = Name + ".Events";
myAECache.QueryEntities = new List<QueryEntity>() { new QueryEntity(typeof(DataItem))
};
myAECache.WriteSynchronizationMode = CacheWriteSynchronizationMode.FullAsync;
myAECache.AtomicityMode = CacheAtomicityMode.Atomic;

ICache<Guid, DataItem> AECache = db.CreateCache<Guid, DataItem>(myAECache);

var ldr = db.GetDataStreamer<Guid, DataItem>(myAECache.Name);


Thanks,

Mario

From: Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
Sent: Thursday, October 19, 2017 16:59
To: user@ignite.apache.org
Subject: Re: Question Ignite Memory Consumption / Object size (Apache Ignite .NET)

Hi Mario,

See https://apacheignite.readme.io/docs/capacity-planning<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite.readme.io%2Fdocs%2Fcapacity-planning&data=02%7C01%7CMElmers%40dow.com%7Cca42f7cc3d3140b4bbfb08d517020f02%7Cc3e32f53cb7f4809968d1cc4ccc785fe%7C0%7C0%7C636440219977766192&sdata=NyeKbTZ06Lqbi8HYZ2uxsj8UfPD22fxp29h3YCI0ACs%3D&reserved=0>

> Each field I have marked with the attribute [QuerySqlField]  and some fields are indexed
This is most likely the case.

1) Have you tried loading data without enabling Ignite SQL (e.g. do not configure CacheConfiguration.QueryEntities)?
2) Can you attach the class? How many fields are there?

Thanks,
Pavel

On Thu, Oct 19, 2017 at 1:57 PM, Elmers, Mario (M) <MElmers@dow.com<mailto:MElmers@dow.com>>
wrote:
Hello,

I try to estimate the needed RAM for my application. I have created 3 nodes by starting only
the Apache.Ignite.exe.

All is done with C# and Apache.Ignite 2.2

Then I created a data loader application which loads up the whole data of my logfiles.

The size of all my logfiles are 7.5 GB. When I loaded it up to the Ignite cluster all together
need more than 32GB of RAM.

My cache is configured as partioned with 0 backups. So I  hat thinked that the cluster will
need not much more than 16 GB of RAM.
Because the file are normal ASCII files which converted to UTF-8 it will twice the amount
of data needed to store.

The class file I have created has for each field of the logentry one field. Each field I have
marked with the attribute [QuerySqlField]  and some fields are indexed.
The key is of type Guid.

Can someone explain why the amount is 4x greater than the raw data ?

Thanks & regards

Mario

Mime
View raw message