Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 85E49200D24 for ; Tue, 24 Oct 2017 15:31:37 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 84783160BE0; Tue, 24 Oct 2017 13:31:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2F45B160BDB for ; Tue, 24 Oct 2017 15:31:35 +0200 (CEST) Received: (qmail 23949 invoked by uid 500); 24 Oct 2017 13:31:33 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 23939 invoked by uid 99); 24 Oct 2017 13:31:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Oct 2017 13:31:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id CCC47C1F6C for ; Tue, 24 Oct 2017 13:31:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.38 X-Spam-Level: ** X-Spam-Status: No, score=2.38 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id hCVWo2L13TQ9 for ; Tue, 24 Oct 2017 13:31:29 +0000 (UTC) Received: from mail-qt0-f180.google.com (mail-qt0-f180.google.com [209.85.216.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 2E9F05F3E1 for ; Tue, 24 Oct 2017 13:31:29 +0000 (UTC) Received: by mail-qt0-f180.google.com with SMTP id z50so30382230qtj.4 for ; Tue, 24 Oct 2017 06:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=oau3fHyIOavyLRs19brO7ZDr3Z7kKJuC4eUgb+oVtMY=; b=lyapm6de2N+srrKP95X4JfvYXH68wCLc5F9YSFTnj3COTRdPjVlpmxXCyUCwFF8QAq BN3dvZ3Es8pqQtz9B2p3C7NfHVOR4SVM9wnHu4O/0aP7L3LcAnd1Ro9k1vvUue8gtODN Uh2HiVL2g9Uto72O0NIR9fuFgZf1Fqte4fGF9BLPugttOu1rNTsSKp8PU7DMU9qELruv 4VS4afOhIWckW13Zx1jgz9MAA0MgBYpw2L9wExPUnUGaohuaFEIrSiehdF13U/USUfxv 9F93/YJcjKPW/Rf2GpPsgmWiHmuRfReMzqneBSUBeaByaLTJ0X4oah8+eFOzjPDIeFCU FkQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=oau3fHyIOavyLRs19brO7ZDr3Z7kKJuC4eUgb+oVtMY=; b=XQb6lzJnMIj0ZShDOcqp6XV4mBkk3bW1Njci8bgjnHfNyNhPLIUCMKflnU3VHDzrip RGH4MXu1AkxL22i+f3yAiKAcxnipDwb1sh8+jWo5hmPfBUCk4E34FhdUjeXr2FoHdg32 48fQtlEC7yYN2TJ8MVfbVM1mZx8TY2EXnXudVRuIdzrMR6vygHaNSmHf4GxaQYEo2wJW WyiKVAUPWkScl/SkbmBJEf+uZZDTlhNjjbt88L31Yj3uEgDqQBLXBaUlDtNkmD5//j4r rZNXzDS/qMyJnz4HPhhfQd9i61+DX+Gv2XJmOL/7b40XLGNmNVHPNs5XknNX3C95tAGZ YKKA== X-Gm-Message-State: AMCzsaV9Cbrq6F/2Dy3fUJt9zxRQ3yshXFBDJs2pSmWGuOfXkvXAkT2N 1xdHVne4JVK0gsaqEzB5lZtkChF6pRRGPVcDNYtZiQ== X-Google-Smtp-Source: ABhQp+SIaH94Cw9QFAAxp48qsKMxEEGwgodJ0dQYbfAzKXxOmRFwzx5/QxFvXv5nrAvCjL159dpPFDt5n2g8zAqFUqw= X-Received: by 10.237.34.28 with SMTP id n28mr25310141qtc.30.1508851883286; Tue, 24 Oct 2017 06:31:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.9.23 with HTTP; Tue, 24 Oct 2017 06:31:22 -0700 (PDT) In-Reply-To: References: From: Evgenii Zhuravlev Date: Tue, 24 Oct 2017 16:31:22 +0300 Message-ID: Subject: Re: Question Ignite Memory Consumption / Object size (Apache Ignite .NET) To: user@ignite.apache.org Content-Type: multipart/alternative; boundary="001a113e4e8685ad7c055c4af410" archived-at: Tue, 24 Oct 2017 13:31:37 -0000 --001a113e4e8685ad7c055c4af410 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mario, You have 7 indexed fields, I think it could be a reason of high memory consumption. As for estimating memory size - I would recommend loading various sized data several times and measuring Memory consumed by it. It will help you to understand approximate needed memory size. These metrics will help you in this: https://apacheignite.readme.io/docs/memory-metrics Regards, Evgenii 2017-10-20 10:35 GMT+03:00 Elmers, Mario (M) : > Hi Pavel, > > > > After doing some research, with QuerySQLField and Indexes, also with the > heap size, > > it seems that with each QuerySQLField the amount of memory needed is > doubled by the size of the > > field. > > > > All test are done by 3 Windows 2012R2 Nodes with Apache.Ignite 2.2 heap > size 1GB per java switch > > 7.5 GB Data ASCII Textfiles > > 64 million rows of data > > Key is a guid data type > > > > So my assumption was that I need a little more than 21GB > > 7.5 GB of ASCII data -> 15GB of UTF dat= a > > 64 million keys of Guid -> 1 GB > > Cache overhead 3 * 300MB -> 1.2 GB > > Static assigned Heap 1GB per Node -> 3 GB > > > -------------------------------- > > Sum > 20.2 GB > > > > test 1 > > data load with No QueryEntities defined. -> 21.4 GB used memory > > > > test 2 > > data load with QueryEntities defined but no [QuerySQLField] attributes. -= > > 22.2 GB used memory > > > > test 3 > > data load with QueryEntities defined and [QuerySQLField] attributes. -> > 38,7 GB used memory > > > > After running test 3 it seems that when I use QuerySQLField then the > memory needed is doubled > > by my estimation. The difference between test 2 and test 3 is 16.5 GB of > memory needed. > > > > src: > > > > DataItem this is the only class which is used. It=E2=80=99s a simple clas= s for > only getting the data into the db. > > > > > > public class DataItem : IBinarizable > > { > > [QuerySqlField(IsIndexed =3D true)] > > public DateTime DateTime; > > [QuerySqlField] > > public short FracSec; > > [QuerySqlField(IsIndexed =3D true)] > > public string EventType =3D ""; > > [QuerySqlField(IsIndexed =3D true)] > > public string Category =3D ""; > > [QuerySqlField(IsIndexed =3D true)] > > public string Area =3D ""; > > [QuerySqlField] > > public string Node =3D ""; > > [QuerySqlField] > > public string Unit =3D ""; > > [QuerySqlField(IsIndexed =3D true)] > > public string Module =3D ""; > > [QuerySqlField] > > public string Module_Description =3D ""; > > [QuerySqlField] > > public string Attribute =3D ""; > > [QuerySqlField(IsIndexed =3D true)] > > public string State =3D ""; > > [QuerySqlField(IsIndexed =3D true)] > > public string Level =3D ""; > > [QuerySqlField] > > public string Desc1 =3D ""; > > [QuerySqlField] > > public string Desc2 =3D ""; > > [QuerySqlField] > > public string Desc3 =3D ""; > > [QuerySqlField] > > public string Desc4 =3D ""; > > > > > > public void ReadBinary(IBinaryReader reader) > > { > > Area =3D reader.ReadString("Area"); > > Attribute =3D reader.ReadString("Attribute"); > > Category =3Dreader.ReadString("Category"); > > DateTime? tmp =3D reader.ReadTimestamp("DateTime"= ); > > if (tmp.HasValue) > > DateTime =3D tmp.Value; > > Desc1 =3Dreader.ReadString("Desc1"); > > Desc2=3Dreader.ReadString("Desc2"); > > Desc3 =3Dreader.ReadString("Desc3"); > > Desc4=3Dreader.ReadString("Desc4"); > > EventType =3Dreader.ReadString("EventType" ); > > FracSec=3D reader.ReadShort("FracSec"); > > Level=3Dreader.ReadString("Level"); > > Module=3Dreader.ReadString("Module"); > > Module_Description=3Dreader.ReadString("Module_Description"); > > Node=3Dreader.ReadString("Node"); > > State=3Dreader.ReadString("State"); > > Unit=3Dreader.ReadString("Unit"); > > } > > > > public void WriteBinary(IBinaryWriter writer) > > { > > writer.WriteString("Area", Area); > > writer.WriteString("Attribute", Attribute); > > writer.WriteString("Category", Category); > > writer.WriteTimestamp("DateTime", DateTime); > > writer.WriteString("Desc1", Desc1); > > writer.WriteString("Desc2", Desc2); > > writer.WriteString("Desc3", Desc3); > > writer.WriteString("Desc4", Desc4); > > writer.WriteString("EventType", EventType); > > writer.WriteShort("FracSec", FracSec); > > writer.WriteString("Level", Level); > > writer.WriteString("Module", Module); > > writer.WriteString("Module_Description", Module_Description); > > writer.WriteString("Node", Node); > > writer.WriteString("State", State); > > writer.WriteString("Unit", Unit); > > } > > } > > > > Apache.Ignite.exe.config addition: > > IgniteConfigurationSection"> > > > > > > > > " maxSize=3D"10737418240"/> > > > > > > > > > > > > DataLoader code cache configuration: > > CacheConfiguration myAECache =3D new CacheConfiguration() ; > > myAECache.CacheMode =3D CacheMode.Partitioned; > > myAECache.WriteThrough =3D false; > > myAECache.ReadThrough =3D false; > > myAECache.Backups =3D 0; > > myAECache.Name =3D Name + ".Events"; > > myAECache.QueryEntities =3D new List() { new QueryEntity(typ= eof(DataItem)) > }; > > myAECache.WriteSynchronizationMode =3D CacheWriteSynchronizationMode. > FullAsync; > > myAECache.AtomicityMode =3D CacheAtomicityMode.Atomic; > > > > ICache AECache =3D db.CreateCache(myAECac= he); > > > > var ldr =3D db.GetDataStreamer(myAECache.Name); > > > > > > Thanks, > > > > Mario > > > > *From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org] > *Sent:* Thursday, October 19, 2017 16:59 > *To:* user@ignite.apache.org > *Subject:* Re: Question Ignite Memory Consumption / Object size (Apache > Ignite .NET) > > > > Hi Mario, > > > > See https://apacheignite.readme.io/docs/capacity-planning > > > > > > Each field I have marked with the attribute [QuerySqlField] and some > fields are indexed > > This is most likely the case. > > > > 1) Have you tried loading data without enabling Ignite SQL (e.g. do not > configure CacheConfiguration.QueryEntities)? > > 2) Can you attach the class? How many fields are there? > > > > Thanks, > > Pavel > > > > On Thu, Oct 19, 2017 at 1:57 PM, Elmers, Mario (M) > wrote: > > Hello, > > > > I try to estimate the needed RAM for my application. I have created 3 > nodes by starting only the Apache.Ignite.exe. > > > > All is done with C# and Apache.Ignite 2.2 > > > > Then I created a data loader application which loads up the whole data of > my logfiles. > > > > The size of all my logfiles are 7.5 GB. When I loaded it up to the Ignite > cluster all together need more than 32GB of RAM. > > > > My cache is configured as partioned with 0 backups. So I hat thinked tha= t > the cluster will need not much more than 16 GB of RAM. > > Because the file are normal ASCII files which converted to UTF-8 it will > twice the amount of data needed to store. > > > > The class file I have created has for each field of the logentry one > field. Each field I have marked with the attribute [QuerySqlField] and > some fields are indexed. > > The key is of type Guid. > > > > Can someone explain why the amount is 4x greater than the raw data ? > > > > Thanks & regards > > > > Mario > > > --001a113e4e8685ad7c055c4af410 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mario,

You have 7 indexed fields, I = think it could be a reason of high memory consumption. As for estimating me= mory size - I would recommend loading various sized data several times and = measuring Memory consumed by it. It will=C2=A0help you to understand approx= imate needed memory size. These metrics will help you in this:=C2=A0https://apacheignit= e.readme.io/docs/memory-metrics

Regards,
=
Evgenii

2017-10-20 10:35 GMT+03:00 Elmers, Mario (M) <<= a href=3D"mailto:MElmers@dow.com" target=3D"_blank">MElmers@dow.com>= :

Hi Pavel,

=C2=A0

After doing some research, with Query= SQLField and Indexes, also with the heap size,

it seems that with each QuerySQLField= the amount of memory needed is doubled by the size of the

field.

=C2=A0

All test are done by 3 Windows 2012R2= Nodes with Apache.Ignite 2.2 heap size 1GB per java switch

7.5 GB Data ASCII Textfiles

64 million rows of data=

Key is a guid data type=

=C2=A0

So my assumption was that I need a li= ttle more than 21GB

7.5 GB of ASCII data =C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 ->=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 15GB of UTF data<= /p>

64 million keys of Guid =C2=A0->= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 =C2=A0=C2=A0=C2=A01 GB

Cache overhead 3 * 300MB =C2=A0->= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.2 GB

Static assigned Heap 1GB per Node -&g= t; =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A03 GB

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ---------------------= -----------

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Sum =C2=A0=C2=A0=C2= =A0=C2=A020.2 GB=C2=A0=C2=A0

=C2=A0

test 1

data load with No QueryEntities defin= ed. -> 21.4 GB used memory

=C2=A0

test 2

data load with QueryEntities defined = but no [QuerySQLField] attributes. -> 22.2 GB used memory<= /span>

=C2=A0

test 3

data load with QueryEntities defined = and [QuerySQLField] attributes. -> 38,7 GB used memory

=C2=A0

After running test 3 it seems that wh= en I use QuerySQLField then the memory needed is doubled

by my estimation. The difference betw= een test 2 and test 3 is 16.5 GB of memory needed.

=C2=A0

src:

=C2=A0

DataItem this is the only class which= is used. It=E2=80=99s a simple class for only getting the data into the db= .

=C2=A0

=C2=A0

=C2=A0=C2=A0=C2=A0 publ= ic clas= s D= ataItem : IBinarizable

=C2=A0=C2=A0=C2=A0 {

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField(IsIndexed =3D true= )]<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic = DateTime DateTime;

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic shor= t F= racSec;

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField(IsIndexed =3D true= )]<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = EventType =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField(IsIndexed =3D true= )]<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Category =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 =C2=A0[QuerySqlField(IsIndexed =3D true= )]<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Area =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Node =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Unit =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField(IsIndexed =3D true= )]<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Module =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Module_Description =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Attribute =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField(IsIndexed =3D true= )]<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = State =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField(IsIndexed =3D true= )]<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Level =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Desc1 =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Desc2 =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Desc3 =3D &= quot;";

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 [QuerySqlField]

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic stri= ng = Desc4 =3D &= quot;";

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0public void= Re= adBinary(IBinaryReader reader)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 {

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Area =3D reader.ReadString("Area&q= uot;);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Attribute =3D reader.ReadString("A= ttribute");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Category =3Dreader.ReadString(<= span style=3D"font-size:9.5pt;font-family:Consolas;color:#a31515">"Cat= egory");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 DateTime? tmp =3D=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 reader.ReadTimestamp(&qu= ot;DateTime");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (tmp= .HasValue)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 DateTime =3D t= mp.Value;

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Desc1 =3Dreader.ReadString("Desc1&= quot;);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Desc2=3Dreader.ReadString("De= sc2");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Desc3 =3Dreader.ReadString("Desc3&= quot;);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Desc4=3Dreader.ReadString("De= sc4");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 EventType =3Dreader.ReadString(= "Ev= entType" );

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 FracSec=3D reader.ReadShort("FracS= ec");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Level=3Dreader.ReadString("Le= vel");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Module=3Dreader.ReadString("M= odule");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Module_Description=3Dreader.ReadS= tring("Module_Description");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Node=3Dreader.ReadString("Node&quo= t;)= ;

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 State=3Dreader.ReadString("St= ate");

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Unit=3Dreader.ReadString("Unit&quo= t;)= ;

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 }

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 publ= ic void= Wr= iteBinary(IBinaryWriter writer)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 {

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Area", Area);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Attribute&quo= t;<= wbr>, Attribute);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Category"= ;, Category);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteTimestamp("DateTi= me", DateTime);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Desc1", Desc1);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Desc2", Desc2);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Desc3", Desc3);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Desc4", Desc4);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("EventType&quo= t;<= wbr>, EventType);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteShort("FracSec"= , FracSec);

=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0writer.WriteString("Level&qu= ot;= , Level);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Module"<= /span>, Module);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Module_D= escription", Module_Description);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Node", Node);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("State", State);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 writer.WriteString("Unit", Unit);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 }

=C2=A0=C2=A0=C2=A0 }

=C2=A0

Apache.Ignite.exe.config addition:

<igniteConfiguration xmlns= =3D<= /span>&quo= t;http://ignite.apache.org/schema/dotnet/IgniteC= onfigurationSection">

=C2=A0=C2=A0=C2=A0 <!--= Customize= Ignite configuration here. --&g= t;<= u>

=C2=A0=C2=A0=C2=A0 <memoryConfi= guration defau= ltMemoryPolicyName=3D"meinTest">

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &l= t;memoryPolicies>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 <memoryPolicyConfiguration name<= /span>=3D"= ;mei= nTest" initi= alSize=3D"5368709120" maxSi= ze= =3D= "10737418240"/> =

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0</memoryPolicies>

=C2=A0=C2=A0=C2=A0 </memoryConf= iguration> =

=C2=A0=C2=A0</igniteConfiguration>

=C2=A0

=C2=A0

DataLoader code cache configuration:<= u>

CacheConfiguration myAECache =3D new<= /span> Cac= heConfiguration() ;

myAECache.CacheMode =3D CacheMod= e.Partitioned;

myAECache.WriteThrough =3D fals= e;<= u>

myAECache.ReadThrough =3D fals= e;<= u>

myAECache.Backups =3D 0;<= u>

myAECache.Name =3D Name + &= quot;.Events";

myAECache.QueryEntities =3D new<= /span> Lis= t<QueryEntity>() { new<= /span> Que= ryEntity(typeof(DataItem)) };

myAECache.WriteSynchronizat= ionMode =3D CacheWriteSynchronizationMode.FullAsync;

myAECache.AtomicityMode =3D Cach= eAtomicityMode.Atomic;

=C2=A0

ICache<Guid, DataItem> AEC= ache =3D db.CreateCache<Guid, DataItem>(myAECache);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0

var ldr =3D db.GetDataStreamer<Guid, DataItem>(myAECache.Nam= e);

=C2=A0

=C2=A0

Thanks,

=C2=A0

Mario

=C2=A0

From: Pavel Tupitsyn [mailto:ptupitsyn@apache.org= ]
Sent: Thursday, October 19, 2017 16:59
To: user= @ignite.apache.org
Subject: Re: Question Ignite Memory Consumption / Object size (Apach= e Ignite .NET)

=C2=A0

Hi Mario,

=C2=A0

=C2=A0

>=C2=A0Each field= I have marked with the attribute [QuerySqlField] =C2=A0and some fields are= indexed

This is most likely = the case.

=C2=A0

1) Have you tried lo= ading data without enabling Ignite SQL (e.g. do not configure CacheConfigur= ation.QueryEntities)?

2) Can you attach th= e class? How many fields are there?

=C2=A0

Thanks,

Pavel<= u>

=C2=A0

On Thu, Oct 19, 2017 at 1:57 PM, Elmers, Mario (M) &= lt;MElmers@dow.com= > wrote:

Hello,

=C2=A0

I try to estimate the needed RAM for my application.= I have created 3 nodes by starting only the Apache.Ignite.exe.

=C2=A0

All is done with C# and Apache.Ignite 2.2<= /u>

=C2=A0

Then I created a data loader application which loads= up the whole data of my logfiles.

=C2=A0

The size of all my logfiles are 7.5 GB. When I loade= d it up to the Ignite cluster all together need more than 32GB of RAM.

=C2=A0

My cache is configured as partioned with 0 backups. = So I =C2=A0hat thinked that the cluster will need not much more than 16 GB = of RAM.

Because the file are normal ASCII files which conver= ted to UTF-8 it will twice the amount of data needed to store.

=C2=A0

The class file I have created has for each field of = the logentry one field. Each field I have marked with the attribute [QueryS= qlField] =C2=A0and some fields are indexed.

The key is of type Guid.

=C2=A0

Can someone explain why the amount is 4x greater tha= n the raw data ?

=C2=A0

Thanks & regards

=C2=A0

Mario

=C2=A0


--001a113e4e8685ad7c055c4af410--