ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Remove cache groups in AI 3.0
Date Tue, 10 Apr 2018 08:41:04 GMT
Vladimir, sounds like a huge refactoring. Other than "cache groups are
confusing", are we solving any other big issues with the new proposed

(every time we try to refactor rebalancing, I get goose bumps)


On Tue, Apr 10, 2018 at 1:32 AM, Vladimir Ozerov <vozerov@gridgain.com>

> Igniters,
> Cache groups were implemented for a sole purpose - to hide internal
> inefficiencies. Namely (add more if I missed something):
> 1) Excessive heap usage for affinity/partition data
> 2) Too much data files as we employ file-per-partition approach.
> These problems were resolved, but now cache groups are a great source of
> confusion both for users and us - hard to understand, no way to configure
> it in deterministic way. Should we resolve mentioned performance issues we
> would never had cache groups. I propose to think we would it take for us to
> get rid of cache groups.
> Please provide your inputs to suggestions below.
> 1) "Merge" partition data from different caches
> Consider that we start a new cache with the same affinity configuration
> (cache mode, partition number, affinity function) as some of already
> existing caches, Is it possible to re-use partition distribution and
> history of existing cache for a new cache? Think of it as a kind of
> automatic cache grouping which is transparent to the user. This would
> remove heap pressure. Also it could resolve our long-standing issue with
> FairAffinityFunction when tow caches with the same affinity configuration
> are not co-located when started on different topology versions.
> 2) Employ segment-extent based approach instead of file-per-partition
> - Every object (cache, index) reside in dedicated segment
> - Segment consists of extents (minimal allocation units)
> - Extents are allocated and deallocated as needed
> - *Ignite specific*: particular extent can be used by only one partition
> - Segments may be located in any number of data files we find convenient
> With this approach "too many fsyncs" problem goes away automatically. At
> the same time it would be possible to implement efficient rebalance still
> as partition data will be split across moderate number of extents, not
> chaotically.
> Once we have p.1 and p.2 ready cache groups could be removed, couldn't
> they?
> Vladimir.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message