From user-return-17938-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Thu Feb 22 00:57:14 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2E92D18061A for ; Thu, 22 Feb 2018 00:57:14 +0100 (CET) Received: (qmail 31658 invoked by uid 500); 21 Feb 2018 23:57:12 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 31648 invoked by uid 99); 21 Feb 2018 23:57:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Feb 2018 23:57:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 78F39C05A5 for ; Wed, 21 Feb 2018 23:57:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.129 X-Spam-Level: ** X-Spam-Status: No, score=2.129 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 6cNIySeMipNs for ; Wed, 21 Feb 2018 23:57:10 +0000 (UTC) Received: from mail-lf0-f48.google.com (mail-lf0-f48.google.com [209.85.215.48]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 9D60A5F397 for ; Wed, 21 Feb 2018 23:57:09 +0000 (UTC) Received: by mail-lf0-f48.google.com with SMTP id l191so4867426lfe.1 for ; Wed, 21 Feb 2018 15:57:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=47JMXnqiihwiYa7wPQZUYGpiVkJ9rPNLzOeTFSeIoDs=; b=KMuIg6f+ZGtNLBpjaYtQCmEHkMYuwSRTs1hzxFosGFxvAUt56LJRwHl03Dv4dC3CfE 1+I1lo9iVdQ2fa9DdCcjJJYdL2pDP2SqlH+pstv8GQIYPh26H72vovntRtONOJmqfZBM mEB+kchzwl2PbznSEuQGScBftuFwVspaFH9ypJH4h2BHg/D6rfSSeHgju3nuteLeEbKy PSbdrMd3FNYZqGWSsdUSgbsM+RQW5yiDZ8tQAlGK6ruhhfmNuCuBlnoD3GQdJX7hbcrz gDQN0o/SRFaZ17RiKo1ovXjKVvpEMvsn16zkXViypi5m4YWHMvpxn7/mYAMssssbKQce ntQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=47JMXnqiihwiYa7wPQZUYGpiVkJ9rPNLzOeTFSeIoDs=; b=eK7PC7ptcVbFuOEpG0tAZS//2p9x/w9jCF2lU2lnORZjmTuUAzWoFZDY8uBa0OcLZ/ ZW606GEeybF+m2U+TLIQYKGNF81t/I9f2VfWkPYeObLlpCdze5mEyzIEmGYDkf7pCe2/ B2EBxiJOmoXe+bPcagmSPmFhPWvUljYwg+Gs23Fa3lY84gfL1eIlRBra3CzJzcfYqf1l 2iJx0P7PfWuRlZAhaol2lysUnJpQBfvgCZzjsm3cBJZUv0b68P9LiJqFF5gzkbjmNBrz cvHuYcNOlTvJntrjleHhVr0J119tKFManKXeyMz11tsbPNH2DYSAhgcKwn1p0LoM1S9q KYBA== X-Gm-Message-State: APf1xPBSre2JlMhyP615/Nkz+xhmJj8eCCf8SZJmnl68SGBiKOUO1Qwi 5h65yM2pDMpm/ZarNY5TuSTYcQ13kH6wmGYhgsE= X-Google-Smtp-Source: AH8x227+RNCIS+mScQ8+m1FRjXoapLpi/1RXHIZEbl1jv0vymW2WsI8Bk92sL/zAduUzvuLWlK7ViUeVg8TPX8DlKZ8= X-Received: by 10.25.162.72 with SMTP id l69mr3808137lfe.38.1519257428664; Wed, 21 Feb 2018 15:57:08 -0800 (PST) MIME-Version: 1.0 References: <1519167496368-0.post@n6.nabble.com> <1519203720628-0.post@n6.nabble.com> In-Reply-To: <1519203720628-0.post@n6.nabble.com> From: Scott Feldstein Date: Wed, 21 Feb 2018 23:56:58 +0000 Message-ID: Subject: Re: tuning near cache performance To: user@ignite.apache.org Content-Type: multipart/alternative; boundary="001a11401c905bb0de0565c1af31" --001a11401c905bb0de0565c1af31 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Roman, 1. Cache Configuration NearCacheConfiguration nearCfg =3D new NearCacheConfiguration<>(); LruEvictionPolicy lruEvictionPolicy =3D new LruEvictionPolicy<>(50000); nearCfg.setNearEvictionPolicy(lruEvictionPolicy); nearCfg.setNearStartSize(50000); CacheConfiguration cacheConfiguration =3D new CacheConfiguration<>(name); cacheConfiguration.setGroupName(=E2=80=9Cmygroup=E2=80=9D); cacheConfiguration.setStatisticsEnabled(true); cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC); cacheConfiguration.setCopyOnRead(false); cacheConfiguration.setEagerTtl(false); return ignite.getOrCreateCache(cacheConfiguration, nearCfg); 2. How many entries in the cache - 50000 3. Cache value example Here is the class: public class Entry { private final long guid; // size of 10-20 per cache private Long[] features; } 4. How large is a cache (GB) - very small, but i'm not sure how i pull that information from ignite (still a newbie). Pre-ignite the cache size was ~ 200MB 5. What is the speed of uploading and downloading (entities per second) Below are the raw stats for one minute of runtime. It is a very read intensive operation. Cluster Cache Metrics CacheMetricsSnapshot [reads=3D2534, puts=3D1365, hits=3D1710, misses=3D824, txCommits=3D0, txRollbacks=3D0, evicts=3D0, remo= ves=3D483, putAvgTimeNanos=3D0.0, getAvgTimeNanos=3D0.0, rmvAvgTimeNanos=3D0.0, commitAvgTimeNanos=3D0.0, rollbackAvgTimeNanos=3D0.0, cacheName=3DtreeLeaderIndex-default, offHeapGets=3D0, offHeapPuts=3D0, offHeapRemoves=3D0, offHeapEvicts=3D0, offHeapHits=3D0, offHeapMisses=3D0, offHeapEntriesCnt=3D819, heapEntriesCnt=3D819, offHeapPrimaryEntriesCnt=3D8= 19, offHeapBackupEntriesCnt=3D0, offHeapAllocatedSize=3D0, size=3D0, keySize=3D= 0, isEmpty=3Dtrue, dhtEvictQueueCurrSize=3D-1, txThreadMapSize=3D0, txXidMapSi= ze=3D0, txCommitQueueSize=3D0, txPrepareQueueSize=3D0, txStartVerCountsSize=3D0, txCommittedVersionsSize=3D0, txRolledbackVersionsSize=3D0, txDhtThreadMapSize=3D0, txDhtXidMapSize=3D-1, txDhtCommitQueueSize=3D0, txDhtPrepareQueueSize=3D0, txDhtStartVerCountsSize=3D0, txDhtCommittedVersionsSize=3D-1, txDhtRolledbackVersionsSize=3D-1, isWriteBehindEnabled=3Dfalse, writeBehindFlushSize=3D-1, writeBehindFlushThreadCnt=3D-1, writeBehindFlushFreq=3D-1, writeBehindStoreBatchSize=3D-1, writeBehindTotalCriticalOverflowCnt=3D-1, writeBehindCriticalOverflowCnt=3D-1, writeBehindErrorRetryCnt=3D-1, writeBehindBufSize=3D-1, totalPartitionsCnt=3D1024, rebalancingPartitionsCn= t=3D0, keysToRebalanceLeft=3D0, rebalancingKeysRate=3D0, rebalancingBytesRate=3D0, rebalanceStartTime=3D0, rebalanceFinishTime=3D0, keyType=3Djava.lang.Object= , valType=3Djava.lang.Object, isStoreByVal=3Dtrue, isStatisticsEnabled=3Dtrue= , isManagementEnabled=3Dfalse, isReadThrough=3Dfalse, isWriteThrough=3Dfalse] cache Local Metrics CacheMetricsSnapshot [reads=3D1747890, puts=3D0, hits=3D1746145, misses=3D1745, txCommits=3D0, txRollbacks=3D0, evicts=3D0, = removes=3D0, putAvgTimeNanos=3D0.0, getAvgTimeNanos=3D12.70161, rmvAvgTimeNanos=3D0.0, commitAvgTimeNanos=3D0.0, rollbackAvgTimeNanos=3D0.0, cacheName=3DtreeLeaderIndex-default, offHeapGets=3D0, offHeapPuts=3D0, offHeapRemoves=3D0, offHeapEvicts=3D0, offHeapHits=3D0, offHeapMisses=3D0, offHeapEntriesCnt=3D0, heapEntriesCnt=3D819, offHeapPrimaryEntriesCnt=3D0, offHeapBackupEntriesCnt=3D0, offHeapAllocatedSize=3D0, size=3D0, keySize=3D= 0, isEmpty=3Dtrue, dhtEvictQueueCurrSize=3D-1, txThreadMapSize=3D0, txXidMapSi= ze=3D0, txCommitQueueSize=3D0, txPrepareQueueSize=3D0, txStartVerCountsSize=3D0, txCommittedVersionsSize=3D0, txRolledbackVersionsSize=3D0, txDhtThreadMapSize=3D0, txDhtXidMapSize=3D-1, txDhtCommitQueueSize=3D0, txDhtPrepareQueueSize=3D0, txDhtStartVerCountsSize=3D0, txDhtCommittedVersionsSize=3D-1, txDhtRolledbackVersionsSize=3D-1, isWriteBehindEnabled=3Dfalse, writeBehindFlushSize=3D-1, writeBehindFlushThreadCnt=3D-1, writeBehindFlushFreq=3D-1, writeBehindStoreBatchSize=3D-1, writeBehindTotalCriticalOverflowCnt=3D-1, writeBehindCriticalOverflowCnt=3D-1, writeBehindErrorRetryCnt=3D-1, writeBehindBufSize=3D-1, totalPartitionsCnt=3D0, rebalancingPartitionsCnt= =3D0, keysToRebalanceLeft=3D0, rebalancingKeysRate=3D0, rebalancingBytesRate=3D0, rebalanceStartTime=3D-1, rebalanceFinishTime=3D-1, keyType=3Djava.lang.Obje= ct, valType=3Djava.lang.Object, isStoreByVal=3Dtrue, isStatisticsEnabled=3Dtrue= , isManagementEnabled=3Dfalse, isReadThrough=3Dfalse, isWriteThrough=3Dfalse] thanks! Scott On Wed, Feb 21, 2018 at 1:02 AM Roman Guseinov wrote: > Hi Scott, > > Performance won't be better than using a local HashMap because there is a > lot of additional procedures for replicating, marshaling and so on. > > In some cases, performance can be improved. Could you provide a more > detailed information? > > 1. Cache configuration > 2. How many entries in the cache > 3. Cache value example > 4. How large is a cache (GB) > 5. What is the speed of uploading and downloading (entities per second) > > Best Regards, > Roman > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > --001a11401c905bb0de0565c1af31 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Roman,
1. Cache Configuration

<= div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 NearCacheConfiguration<K, V> nea= rCfg =3D new NearCacheConfiguration<>();
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 LruEvictionPolicy<K, V> lruEvictionPolicy =3D new LruEvict= ionPolicy<>(50000);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 nearCfg.set= NearEvictionPolicy(lruEvictionPolicy);
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 nearCfg.setNearStartSize(50000);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = CacheConfiguration<K, V> cacheConfiguration =3D new CacheConfiguratio= n<>(name);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 cacheConfiguration.s= etGroupName(=E2=80=9Cmygroup=E2=80=9D);
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 cacheConfiguration.setStatisticsEnabled(true);
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 cacheConfiguration.setAtomicityMode(CacheA= tomicityMode.ATOMIC);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 cacheConfigurat= ion.setCopyOnRead(false);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 cacheConfig= uration.setEagerTtl(false);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 return ig= nite.getOrCreateCache(cacheConfiguration, nearCfg);

2. How many entries in the cache - 50000
3. Cache value e= xample
Here is the class:
=C2=A0 =C2=A0 public cla= ss Entry<Long> {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 private final = long guid;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 // size of 10-20 per cache=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 private Long[] features;
= =C2=A0 =C2=A0 }
4. How large is a cache (GB) - very small, = but i'm not sure how i pull that information from ignite (still a newbi= e).=C2=A0 Pre-ignite the cache size was ~ 200MB
5. What is the sp= eed of uploading and downloading (entities per second)
Below are = the raw stats for one minute of runtime.=C2=A0 It is a very read intensive = operation.

Cluster Cache Metrics CacheMetrics= Snapshot [reads=3D2534, puts=3D1365, hits=3D1710, misses=3D824, txCommits= =3D0, txRollbacks=3D0, evicts=3D0, removes=3D483, putAvgTimeNanos=3D0.0, ge= tAvgTimeNanos=3D0.0, rmvAvgTimeNanos=3D0.0, commitAvgTimeNanos=3D0.0, rollb= ackAvgTimeNanos=3D0.0, cacheName=3DtreeLeaderIndex-default, offHeapGets=3D0= , offHeapPuts=3D0, offHeapRemoves=3D0, offHeapEvicts=3D0, offHeapHits=3D0, = offHeapMisses=3D0, offHeapEntriesCnt=3D819, heapEntriesCnt=3D819, offHeapPr= imaryEntriesCnt=3D819, offHeapBackupEntriesCnt=3D0, offHeapAllocatedSize=3D= 0, size=3D0, keySize=3D0, isEmpty=3Dtrue, dhtEvictQueueCurrSize=3D-1, txThr= eadMapSize=3D0, txXidMapSize=3D0, txCommitQueueSize=3D0, txPrepareQueueSize= =3D0, txStartVerCountsSize=3D0, txCommittedVersionsSize=3D0, txRolledbackVe= rsionsSize=3D0, txDhtThreadMapSize=3D0, txDhtXidMapSize=3D-1, txDhtCommitQu= eueSize=3D0, txDhtPrepareQueueSize=3D0, txDhtStartVerCountsSize=3D0, txDhtC= ommittedVersionsSize=3D-1, txDhtRolledbackVersionsSize=3D-1, isWriteBehindE= nabled=3Dfalse, writeBehindFlushSize=3D-1, writeBehindFlushThreadCnt=3D-1, = writeBehindFlushFreq=3D-1, writeBehindStoreBatchSize=3D-1, writeBehindTotal= CriticalOverflowCnt=3D-1, writeBehindCriticalOverflowCnt=3D-1, writeBehindE= rrorRetryCnt=3D-1, writeBehindBufSize=3D-1, totalPartitionsCnt=3D1024, reba= lancingPartitionsCnt=3D0, keysToRebalanceLeft=3D0, rebalancingKeysRate=3D0,= rebalancingBytesRate=3D0, rebalanceStartTime=3D0, rebalanceFinishTime=3D0,= keyType=3Djava.lang.Object, valType=3Djava.lang.Object, isStoreByVal=3Dtru= e, isStatisticsEnabled=3Dtrue, isManagementEnabled=3Dfalse, isReadThrough= =3Dfalse, isWriteThrough=3Dfalse]

cache Local Metr= ics CacheMetricsSnapshot [reads=3D1747890, puts=3D0, hits=3D1746145, misses= =3D1745, txCommits=3D0, txRollbacks=3D0, evicts=3D0, removes=3D0, putAvgTim= eNanos=3D0.0, getAvgTimeNanos=3D12.70161, rmvAvgTimeNanos=3D0.0, commitAvgT= imeNanos=3D0.0, rollbackAvgTimeNanos=3D0.0, cacheName=3DtreeLeaderIndex-def= ault, offHeapGets=3D0, offHeapPuts=3D0, offHeapRemoves=3D0, offHeapEvicts= =3D0, offHeapHits=3D0, offHeapMisses=3D0, offHeapEntriesCnt=3D0, heapEntrie= sCnt=3D819, offHeapPrimaryEntriesCnt=3D0, offHeapBackupEntriesCnt=3D0, offH= eapAllocatedSize=3D0, size=3D0, keySize=3D0, isEmpty=3Dtrue, dhtEvictQueueC= urrSize=3D-1, txThreadMapSize=3D0, txXidMapSize=3D0, txCommitQueueSize=3D0,= txPrepareQueueSize=3D0, txStartVerCountsSize=3D0, txCommittedVersionsSize= =3D0, txRolledbackVersionsSize=3D0, txDhtThreadMapSize=3D0, txDhtXidMapSize= =3D-1, txDhtCommitQueueSize=3D0, txDhtPrepareQueueSize=3D0, txDhtStartVerCo= untsSize=3D0, txDhtCommittedVersionsSize=3D-1, txDhtRolledbackVersionsSize= =3D-1, isWriteBehindEnabled=3Dfalse, writeBehindFlushSize=3D-1, writeBehind= FlushThreadCnt=3D-1, writeBehindFlushFreq=3D-1, writeBehindStoreBatchSize= =3D-1, writeBehindTotalCriticalOverflowCnt=3D-1, writeBehindCriticalOverflo= wCnt=3D-1, writeBehindErrorRetryCnt=3D-1, writeBehindBufSize=3D-1, totalPar= titionsCnt=3D0, rebalancingPartitionsCnt=3D0, keysToRebalanceLeft=3D0, reba= lancingKeysRate=3D0, rebalancingBytesRate=3D0, rebalanceStartTime=3D-1, reb= alanceFinishTime=3D-1, keyType=3Djava.lang.Object, valType=3Djava.lang.Obje= ct, isStoreByVal=3Dtrue, isStatisticsEnabled=3Dtrue, isManagementEnabled=3D= false, isReadThrough=3Dfalse, isWriteThrough=3Dfalse]

<= /div>
thanks!

Scott

<= br>

On Wed, Feb = 21, 2018 at 1:02 AM Roman Guseinov <roman@gromtech.ru> wrote:
H= i Scott,

Performance won't be better than using a local HashMap because there is= a
lot of additional procedures for replicating, marshaling and so on.

In some cases, performance can be improved. Could you provide a more
detailed information?

1. Cache configuration
2. How many entries in the cache
3. Cache value example
4. How large is a cache (GB)
5. What is the speed of uploading and downloading (entities per second)

Best Regards,
Roman



--
Sent from: http://apache-ignite-users.70518.x6.nabbl= e.com/
--001a11401c905bb0de0565c1af31--