ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Scherbakov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-10920) Optimize HistoryAffinityAssignment heap usage.
Date Thu, 31 Jan 2019 19:19:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-10920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757648#comment-16757648
] 

Alexei Scherbakov commented on IGNITE-10920:
--------------------------------------------

[~kbolyandra],

I've played a bit with JOL benchmark and got following results for a single history assignment
with 32 and 128 nodes in topology, results are impressive:

{{Heap usage [optimized=false, parts=32768, nodeCnt=32, backups=2, footprint:
 COUNT AVG SUM DESCRIPTION
 33714 39 1340960 [Ljava.lang.Object;
 33714 24 809136 java.util.ArrayList
 2 24 48 java.util.Collections$UnmodifiableRandomAccessList
 1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
 1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
 67432 2150208 (total)

]
Heap usage [optimized=true, parts=32768, nodeCnt=32, backups=2, footprint:
 COUNT AVG SUM DESCRIPTION
 945 232 219280 [C
 1 8208 8208 [Ljava.util.HashMap$Node;
 1 144 144 [Lorg.apache.ignite.cluster.ClusterNode;
 944 16 15104 java.lang.Integer
 1 48 48 java.util.HashMap
 944 32 30208 java.util.HashMap$Node
 1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
 1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
 1 32 32 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$1
 1 32 32 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$2
 2840 273120 (total)

]
Optimization: optimized=273120, deoptimized=2150208 rate: 7.872}}

{{Heap usage [optimized=false, parts=32768, nodeCnt=128, backups=2, footprint:
 COUNT AVG SUM DESCRIPTION
 33066 39 1320224 [Ljava.lang.Object;
 33066 24 793584 java.util.ArrayList
 2 24 48 java.util.Collections$UnmodifiableRandomAccessList
 1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
 1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
 66136 2113920 (total)

]
Heap usage [optimized=true, parts=32768, nodeCnt=128, backups=2, footprint:
 COUNT AVG SUM DESCRIPTION
 297 685 203728 [C
 1 2064 2064 [Ljava.util.HashMap$Node;
 1 528 528 [Lorg.apache.ignite.cluster.ClusterNode;
 296 16 4736 java.lang.Integer
 1 48 48 java.util.HashMap
 296 32 9472 java.util.HashMap$Node
 1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
 1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
 1 32 32 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$1
 1 32 32 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$2
 896 220704 (total)

Optimization: optimized=220704, deoptimized=2113920 rate: 9.578}}

No objections from my side.

I think somebody of the commiters will pass by shortly and finish your contribution.

 

 

> Optimize HistoryAffinityAssignment heap usage.
> ----------------------------------------------
>
>                 Key: IGNITE-10920
>                 URL: https://issues.apache.org/jira/browse/IGNITE-10920
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Alexei Scherbakov
>            Assignee: Konstantin Bolyandra
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> With large topology and large amount of caches/partitions many server discovery events
may quickly produce large affinity history, eating gigabytes of heap.
> Solution: implement some kind of a compression for affinity cache map.
> On example, affinity history could be stored as delta to some previous version.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message