Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7F4A3200C1A for ; Mon, 13 Feb 2017 14:16:16 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 7DC09160B60; Mon, 13 Feb 2017 13:16:16 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2B244160B4D for ; Mon, 13 Feb 2017 14:16:15 +0100 (CET) Received: (qmail 73691 invoked by uid 500); 13 Feb 2017 13:16:14 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 73679 invoked by uid 99); 13 Feb 2017 13:16:14 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Feb 2017 13:16:14 +0000 Received: from mail-ot0-f171.google.com (mail-ot0-f171.google.com [74.125.82.171]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id DE2971A00A2 for ; Mon, 13 Feb 2017 13:16:13 +0000 (UTC) Received: by mail-ot0-f171.google.com with SMTP id 65so66717733otq.2 for ; Mon, 13 Feb 2017 05:16:13 -0800 (PST) X-Gm-Message-State: AMke39mJxZFImK8T3wRXvaQJrQJL2fsVHBo70uuyv+0AbqmsxKalCZvmz+2YcjtejaKSXhoU6e4GBx73uIC4drYh X-Received: by 10.157.7.53 with SMTP id 50mr11427790ote.91.1486991773284; Mon, 13 Feb 2017 05:16:13 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.54.129 with HTTP; Mon, 13 Feb 2017 05:15:42 -0800 (PST) In-Reply-To: References: From: Pavel Tupitsyn Date: Mon, 13 Feb 2017 16:15:42 +0300 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Effective size limit for cache items in Ignite To: user@ignite.apache.org Content-Type: multipart/alternative; boundary=001a113f52d06e53f505486940f9 archived-at: Mon, 13 Feb 2017 13:16:16 -0000 --001a113f52d06e53f505486940f9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Raymond, CopyOnRead setting has no effect in .NET code. It exists for cases when there are both Java and .NET nodes in a cluster. In Ignite.NET each cache.Get call causes two things: 1) Copy a piece of memory from Java (serialized data) 2) Deserialize the data into user type instance. You can avoid deserialization by using Binary Mode: https://apacheignite-net.readme.io/docs/binary-mode Pavel On Mon, Feb 13, 2017 at 6:38 AM, Raymond Wilson wrote: > I found this code snippet in the Ignite v1.8 source: > > > > This appears to be the core place where the value of CopyOnRead has an > effect, though I don=E2=80=99t understand the context of the other logica= l > conditions around it; it seems that CopyOnRead is dependent on other > configuration state before it will be honoured. I=E2=80=99m still digging= to locate > the detailed help around this setting to understand these other condition= s. > > /** {@inheritDoc} */ > > @Override public CacheObjectContext contextForCache(CacheConfiguratio= n > ccfg) throws IgniteCheckedException { > > assert ccfg !=3D null; > > > > CacheMemoryMode memMode =3D ccfg.getMemoryMode(); > > > > boolean storeVal =3D !ccfg.isCopyOnRead() || (!isBinaryEnabled(cc= fg) > && > > (GridQueryProcessor.isEnabled(ccfg) || ctx.config(). > isPeerClassLoadingEnabled())); > > > > CacheObjectContext res =3D new CacheObjectContext(ctx, > > ccfg.getName(), > > ccfg.getAffinityMapper() !=3D null ? ccfg.getAffinityMapper()= : > new GridCacheDefaultAffinityKeyMapper(), > > ccfg.isCopyOnRead() && memMode !=3D OFFHEAP_VALUES, > > storeVal, > > ctx.config().isPeerClassLoadingEnabled() && > !isBinaryEnabled(ccfg)); > > > > ctx.resource().injectGeneric(res.defaultAffMapper()); > > > > return res; > > } > > > > Thanks, > > Raymond. > > > > *From:* Raymond Wilson [mailto:raymond_wilson@trimble.com] > *Sent:* Monday, February 13, 2017 2:43 PM > *To:* user@ignite.apache.org > *Subject:* RE: Effective size limit for cache items in Ignite > > > > Ah, I found the CopyOnRead flag in the cache configuration. > > > > Unfortunately, it seems to have the same behaviour regardless of the > setting for this flag. > > > > If I create an example like the below, it seems that querying the same > element from the cache many times takes about the same amount of time in > both cases. Visual Studio also reports large numbers of GC episodes while > it cleans up the large freed MyCacheClass instances. Is this flag only > applicable to Java contexts? I did also try setting KeepBinaryInStore to > true, though there was no noticeable difference. > > > > [Serializable] > > public class MyCacheClass > > { > > public String name =3D String.Empty; > > private byte[] localData =3D null; > > > > public MyCacheClass(String _name) > > { > > name =3D _name; > > localData =3D new byte[4000000]; > > } > > } > > > > class Program > > { > > static void Main(string[] args) > > { > > IIgnite ignite =3D Ignition.Start(); > > > > // Add a cache to Ignite > > ICache cache =3D ignite.CreateCache MyCacheClass> > > (new CacheConfiguration() > > { > > Name =3D "TestCache", > > CopyOnRead =3D false, > > KeepBinaryInStore =3D true > > }); > > > > // Add a cache item > > cache.Put("First", new MyCacheClass("FirstItem")); > > > > // query back the cache items > > for (int i =3D 0; i < 30000; i++) > > { > > MyCacheClass first =3D cache.Get("First"); > > } > > } > > > > > > *From:* Raymond Wilson [mailto:raymond_wilson@trimble.com > ] > *Sent:* Monday, February 13, 2017 11:35 AM > *To:* user@ignite.apache.org > *Subject:* Effective size limit for cache items in Ignite > > > > Hi, > > > > What is the practical size limit for items in an Ignite cache? > > > > I suspect the answer is something =E2=80=9CAs large as the memory you hav= e to hold > it=E2=80=9D, but my question is more aimed at the practicality of large i= tems in a > cache due to the overhead of pulling copies of the items out of the cache > in response to a Cache.Get() request. > > > > For instance, let=E2=80=99s say I had cache items in the 64Kb size range,= and had > requests that commonly refer to those cache items to perform some work on > them in response to a request. Will each Cache.Get() request require an > extraction and repackaging of the cache item prior to handing it back to > the caller as a new (copied) version of that cache item, or is there a wa= y > for just a reference to the cache item to be returned to the caller? > > > > I understand there is a way to designate the information in a cache as > just blobs of data with no serialisation semantics. In this case does a > Cache.Get() return a pointer or a copy (with a local locking semantic to > prevent change)? > > > > Thanks, > > Raymond. > > > --001a113f52d06e53f505486940f9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Raymond,

CopyOnRead setting has no e= ffect in .NET code.
It exists for cases when there are both Java = and .NET nodes in a cluster.

In Ignite.NET each ca= che.Get call causes two things:
1) Copy a piece of memory from Ja= va (serialized data)
2) Deserialize the data into user type insta= nce.

You can avoid deserialization by using Binary= Mode:

Pavel

On Mon, Feb 13, 2017 at 6:38 AM, Raymond Wilson <raymond_wilson@trimble.com> wrote:

I found this code snippet in the Ignite v1.8 source:

=C2=A0

This appears to be the cor= e place where the value of CopyOnRead has an effect, though I don=E2=80=99t= understand the context of the other logical conditions around it; it seems= that CopyOnRead is dependent on other configuration state before it will b= e honoured. I=E2=80=99m still digging to locate the detailed help around th= is setting to understand these other conditions.

=C2=A0=C2=A0=C2=A0=C2=A0/** {@inheritDoc} */

=C2=A0=C2=A0=C2= =A0 @Override public CacheObjectContext contextForCache(CacheConfigura= tion ccfg) throws IgniteCheckedException {

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a= ssert ccfg !=3D null;

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CacheMemoryMode memMode =3D= ccfg.getMemoryMode();

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 boolean storeVal =3D !ccfg= .isCopyOnRead() || (!isBinaryEnabled(ccfg) &&

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (GridQueryProcessor.isEnabled(ccfg) = || ctx.config().isPeerClassLoadingEnabled()));

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CacheObjectContext res =3D new CacheObjectContext(ctx,

=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ccfg.getName(),

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ccfg.getAffinityMapper() !=3D null ? c= cfg.getAffinityMapper() : new GridCacheDefaultAffinityKeyMapper(),

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ccfg.isCopyOnRead() &am= p;& memMode !=3D OFFHEAP_VALUES,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 storeVal,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 ctx.config().isPeerClassLoadingEnabled() && !isBi= naryEnabled(ccfg));

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ctx.resource().injectGeneric(= res.defaultAffMapper());

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return res;=

=C2=A0=C2=A0=C2=A0= }

=C2=A0

Thanks,=

Raymond.

=C2=A0

From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Monday,= February 13, 2017 2:43 PM
To: user@ignite.apache.org
Subject: RE:= Effective size limit for cache items in Ignite

=

=C2=A0

<= span style=3D"color:#1f497d">Ah, I found the CopyOnRead flag in the cache c= onfiguration.

=C2=A0

Un= fortunately, it seems to have the same behaviour regardless of the setting = for this flag.

=C2=A0

If= I create an example like the below, it seems that querying the same elemen= t from the cache many times takes about the same amount of time in both cas= es. Visual Studio also reports large numbers of GC episodes while it cleans= up the large freed MyCacheClass instances. Is this flag only applicable to= Java contexts? I did also try setting KeepBinaryInStore to true, though th= ere was no noticeable difference.

=C2=A0

[Serializable]

=C2= =A0=C2=A0=C2=A0 public class MyCacheClass

=C2=A0=C2=A0=C2=A0 {

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 public String name =3D String.Empty;

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 private byte[] localData =3D null;

=C2=A0

= =C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 public MyCacheClass(String _name)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 {

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 name =3D _name;<= /p>

=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 localData =3D new byte[4000000];

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 }

=C2=A0=C2=A0= =C2=A0 }

=C2=A0=

=C2=A0=C2=A0=C2=A0 class<= span style=3D"font-size:9.5pt;font-family:Consolas;color:black"> Program=

=C2=A0=C2=A0=C2=A0 {<= /p>

=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 static void Main(string[] args)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 {

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 IIgnite ignite =3D Ignition.Start();

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 // Add a cache to Ignite

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ICach= e&l= t;String, MyCacheClass> cache =3D ignite.CreateCache<String, MyCacheClass= >

=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (new CacheConfigura= tion()

=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 {<= /span>

=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 Name =3D "TestCache",

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CopyOnRe= ad =3D false,

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 KeepBinaryInStore =3D true

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 });

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <= /span>// A= dd a cache item

=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cache.Put(= "Fi= rst", new MyCacheClass("FirstItem"));

=C2=A0

= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <= span style=3D"font-size:9.5pt;font-family:Consolas;color:green">// query ba= ck the cache items

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for (int i =3D 0; i < 300= 00; i++)

=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 {

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 MyCacheClass first =3D cach= e.Get("First");

= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }=

=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 }

=C2= =A0

=C2=A0

From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent= : Monday, February 13, 2017 11:35 AM
To: user@ignite.apache.org
Su= bject: Effective size limit for cache items in Ignite

<= /div>

=C2=A0

Hi,

=C2=A0

What is the practical si= ze limit for items in an Ignite cache?

=C2=A0

I suspect the answer is something =E2=80=9CAs large as the memory you = have to hold it=E2=80=9D, but my question is more aimed at the practicality= of large items in a cache due to the overhead of pulling copies of the ite= ms out of the cache in response to a Cache.Get() request.

=C2=A0

For instance, let=E2=80=99s say I had cache items in= the 64Kb size range, and had requests that commonly refer to those cache i= tems to perform some work on them in response to a request. Will each Cache= .Get() request require an extraction and repackaging of the cache item prio= r to handing it back to the caller as a new (copied) version of that cache = item, or is there a way for just a reference to the cache item to be return= ed to the caller?

=C2= =A0

I understand ther= e is a way to designate the information in a cache as just blobs of data wi= th no serialisation semantics. In this case does a Cache.Get() return a poi= nter or a copy (with a local locking semantic to prevent change)?

=C2=A0

Thanks,

Raymond.

=C2=A0


--001a113f52d06e53f505486940f9--