ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@gridgain.com>
Subject Re: Ideal number of partitions for a small cache
Date Tue, 22 Dec 2015 10:24:39 GMT
Hi Noam,

The main rule is that a number of partitions should be bigger than a 
number of the nodes in a cluster.
Otherwise you definitely won't get uniform number of partitions' 
assignments across all the cluster nodes.

In practice the rendezvous affinity function sometimes has ~10% 
inaccuracy meaning that some nodes will have more partitions than the other.
So in general I would make sure that the number of partitions is at 
least two times bigger than the number of nodes.
However, I haven't talked to anyone who changed the default number of 
1024 to another one value.

At the implementation layer a partition is a ConcurrentHashMap. So 
whether you have less or more such maps the overhead shouldn't be 
significant.

However, bigger number of partitions has its own advantages over a 
smaller number. As an example if you want to perform a scan query over 
all partitions you can split this task to a number of cores you have and 
iterate over many partitions in parallel.

Regards,
Denis

On 12/22/2015 12:25 PM, Noam Liran wrote:
> Hi,
>
> We’re using RendezvousAffinityFunction as our cache affinity function 
> and were wondering –
> how can we determine the ideal number of partitions for a small cache?
>
> The default value is 1024 which seems a bit high for a small cluster; 
> Since we’re not sure
> what’s the overheard of each partition, we’re not sure what a good 
> number would be.
>
> Would appreciate any suggestions.
>
> Regards,
> Noam
>


Mime
View raw message