ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evgenii Zhuravlev <e.zhuravlev...@gmail.com>
Subject Re: Question Ignite Memory Consumption / Object size (Apache Ignite .NET)
Date Tue, 24 Oct 2017 13:31:22 GMT
Hi Mario,

You have 7 indexed fields, I think it could be a reason of high memory
consumption. As for estimating memory size - I would recommend loading
various sized data several times and measuring Memory consumed by it. It
will help you to understand approximate needed memory size. These metrics
will help you in this: https://apacheignite.readme.io/docs/memory-metrics

Regards,
Evgenii

2017-10-20 10:35 GMT+03:00 Elmers, Mario (M) <MElmers@dow.com>:

> Hi Pavel,
>
>
>
> After doing some research, with QuerySQLField and Indexes, also with the
> heap size,
>
> it seems that with each QuerySQLField the amount of memory needed is
> doubled by the size of the
>
> field.
>
>
>
> All test are done by 3 Windows 2012R2 Nodes with Apache.Ignite 2.2 heap
> size 1GB per java switch
>
> 7.5 GB Data ASCII Textfiles
>
> 64 million rows of data
>
> Key is a guid data type
>
>
>
> So my assumption was that I need a little more than 21GB
>
> 7.5 GB of ASCII data        ->                            15GB of UTF data
>
> 64 million keys of Guid  ->                               1 GB
>
> Cache overhead 3 * 300MB  ->                   1.2 GB
>
> Static assigned Heap 1GB per Node ->       3 GB
>
>
> --------------------------------
>
>                                                                 Sum
>     20.2 GB
>
>
>
> test 1
>
> data load with No QueryEntities defined. -> 21.4 GB used memory
>
>
>
> test 2
>
> data load with QueryEntities defined but no [QuerySQLField] attributes. ->
> 22.2 GB used memory
>
>
>
> test 3
>
> data load with QueryEntities defined and [QuerySQLField] attributes. ->
> 38,7 GB used memory
>
>
>
> After running test 3 it seems that when I use QuerySQLField then the
> memory needed is doubled
>
> by my estimation. The difference between test 2 and test 3 is 16.5 GB of
> memory needed.
>
>
>
> src:
>
>
>
> DataItem this is the only class which is used. It’s a simple class for
> only getting the data into the db.
>
>
>
>
>
>     public class DataItem : IBinarizable
>
>     {
>
>         [QuerySqlField(IsIndexed = true)]
>
>         public DateTime DateTime;
>
>         [QuerySqlField]
>
>         public short FracSec;
>
>         [QuerySqlField(IsIndexed = true)]
>
>         public string EventType = "";
>
>         [QuerySqlField(IsIndexed = true)]
>
>         public string Category = "";
>
>         [QuerySqlField(IsIndexed = true)]
>
>         public string Area = "";
>
>         [QuerySqlField]
>
>         public string Node = "";
>
>         [QuerySqlField]
>
>         public string Unit = "";
>
>         [QuerySqlField(IsIndexed = true)]
>
>         public string Module = "";
>
>         [QuerySqlField]
>
>         public string Module_Description = "";
>
>         [QuerySqlField]
>
>         public string Attribute = "";
>
>         [QuerySqlField(IsIndexed = true)]
>
>         public string State = "";
>
>         [QuerySqlField(IsIndexed = true)]
>
>         public string Level = "";
>
>         [QuerySqlField]
>
>         public string Desc1 = "";
>
>         [QuerySqlField]
>
>         public string Desc2 = "";
>
>         [QuerySqlField]
>
>         public string Desc3 = "";
>
>         [QuerySqlField]
>
>         public string Desc4 = "";
>
>
>
>
>
>         public void ReadBinary(IBinaryReader reader)
>
>         {
>
>             Area = reader.ReadString("Area");
>
>             Attribute = reader.ReadString("Attribute");
>
>             Category =reader.ReadString("Category");
>
>             DateTime? tmp =             reader.ReadTimestamp("DateTime");
>
>             if (tmp.HasValue)
>
>                 DateTime = tmp.Value;
>
>             Desc1 =reader.ReadString("Desc1");
>
>             Desc2=reader.ReadString("Desc2");
>
>             Desc3 =reader.ReadString("Desc3");
>
>             Desc4=reader.ReadString("Desc4");
>
>             EventType =reader.ReadString("EventType" );
>
>             FracSec= reader.ReadShort("FracSec");
>
>             Level=reader.ReadString("Level");
>
>             Module=reader.ReadString("Module");
>
>             Module_Description=reader.ReadString("Module_Description");
>
>             Node=reader.ReadString("Node");
>
>             State=reader.ReadString("State");
>
>             Unit=reader.ReadString("Unit");
>
>         }
>
>
>
>         public void WriteBinary(IBinaryWriter writer)
>
>         {
>
>             writer.WriteString("Area", Area);
>
>             writer.WriteString("Attribute", Attribute);
>
>             writer.WriteString("Category", Category);
>
>             writer.WriteTimestamp("DateTime", DateTime);
>
>             writer.WriteString("Desc1", Desc1);
>
>             writer.WriteString("Desc2", Desc2);
>
>             writer.WriteString("Desc3", Desc3);
>
>             writer.WriteString("Desc4", Desc4);
>
>             writer.WriteString("EventType", EventType);
>
>             writer.WriteShort("FracSec", FracSec);
>
>             writer.WriteString("Level", Level);
>
>             writer.WriteString("Module", Module);
>
>             writer.WriteString("Module_Description", Module_Description);
>
>             writer.WriteString("Node", Node);
>
>             writer.WriteString("State", State);
>
>             writer.WriteString("Unit", Unit);
>
>         }
>
>     }
>
>
>
> Apache.Ignite.exe.config addition:
>
> <igniteConfiguration xmlns="http://ignite.apache.org/schema/dotnet/
> IgniteConfigurationSection">
>
>     <!-- Customize Ignite configuration here. -->
>
>     <memoryConfiguration defaultMemoryPolicyName="meinTest">
>
>       <memoryPolicies>
>
>         <memoryPolicyConfiguration name="meinTest" initialSize="5368709120
> " maxSize="10737418240"/>
>
>       </memoryPolicies>
>
>     </memoryConfiguration>
>
>   </igniteConfiguration>
>
>
>
>
>
> DataLoader code cache configuration:
>
> CacheConfiguration myAECache = new CacheConfiguration() ;
>
> myAECache.CacheMode = CacheMode.Partitioned;
>
> myAECache.WriteThrough = false;
>
> myAECache.ReadThrough = false;
>
> myAECache.Backups = 0;
>
> myAECache.Name = Name + ".Events";
>
> myAECache.QueryEntities = new List<QueryEntity>() { new QueryEntity(typeof(DataItem))
> };
>
> myAECache.WriteSynchronizationMode = CacheWriteSynchronizationMode.
> FullAsync;
>
> myAECache.AtomicityMode = CacheAtomicityMode.Atomic;
>
>
>
> ICache<Guid, DataItem> AECache = db.CreateCache<Guid, DataItem>(myAECache);
>
>
>
> var ldr = db.GetDataStreamer<Guid, DataItem>(myAECache.Name);
>
>
>
>
>
> Thanks,
>
>
>
> Mario
>
>
>
> *From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
> *Sent:* Thursday, October 19, 2017 16:59
> *To:* user@ignite.apache.org
> *Subject:* Re: Question Ignite Memory Consumption / Object size (Apache
> Ignite .NET)
>
>
>
> Hi Mario,
>
>
>
> See https://apacheignite.readme.io/docs/capacity-planning
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite.readme.io%2Fdocs%2Fcapacity-planning&data=02%7C01%7CMElmers%40dow.com%7Cca42f7cc3d3140b4bbfb08d517020f02%7Cc3e32f53cb7f4809968d1cc4ccc785fe%7C0%7C0%7C636440219977766192&sdata=NyeKbTZ06Lqbi8HYZ2uxsj8UfPD22fxp29h3YCI0ACs%3D&reserved=0>
>
>
>
> > Each field I have marked with the attribute [QuerySqlField]  and some
> fields are indexed
>
> This is most likely the case.
>
>
>
> 1) Have you tried loading data without enabling Ignite SQL (e.g. do not
> configure CacheConfiguration.QueryEntities)?
>
> 2) Can you attach the class? How many fields are there?
>
>
>
> Thanks,
>
> Pavel
>
>
>
> On Thu, Oct 19, 2017 at 1:57 PM, Elmers, Mario (M) <MElmers@dow.com>
> wrote:
>
> Hello,
>
>
>
> I try to estimate the needed RAM for my application. I have created 3
> nodes by starting only the Apache.Ignite.exe.
>
>
>
> All is done with C# and Apache.Ignite 2.2
>
>
>
> Then I created a data loader application which loads up the whole data of
> my logfiles.
>
>
>
> The size of all my logfiles are 7.5 GB. When I loaded it up to the Ignite
> cluster all together need more than 32GB of RAM.
>
>
>
> My cache is configured as partioned with 0 backups. So I  hat thinked that
> the cluster will need not much more than 16 GB of RAM.
>
> Because the file are normal ASCII files which converted to UTF-8 it will
> twice the amount of data needed to store.
>
>
>
> The class file I have created has for each field of the logentry one
> field. Each field I have marked with the attribute [QuerySqlField]  and
> some fields are indexed.
>
> The key is of type Guid.
>
>
>
> Can someone explain why the amount is 4x greater than the raw data ?
>
>
>
> Thanks & regards
>
>
>
> Mario
>
>
>

Mime
View raw message