ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Sapego <isap...@apache.org>
Subject Re: Best Effort Affinity for thin clients
Date Mon, 04 Feb 2019 13:44:36 GMT
Ok, I understand now. I'll try updating IEP according to this proposal and
notify you guys.

Best Regards,
Igor


On Mon, Feb 4, 2019 at 4:27 PM Vladimir Ozerov <vozerov@gridgain.com> wrote:

> Igor,
>
> My idea is simply to add the list of caches with the same distribution to
> the end of partition response. Client can use this information to populate
> partition info for more caches in a single request.
>
> On Mon, Feb 4, 2019 at 3:06 PM Igor Sapego <isapego@apache.org> wrote:
>
> > Vladimir,
> >
> > So correct me if I'm wrong, what you propose is to avoid mentioning
> > of cache groups, and use instead of "cache group" term something like
> > "distribution"? Or do you propose some changes in protocol? If so, can
> > you briefly explain, what kind of changes they are?
> >
> > Best Regards,
> > Igor
> >
> >
> > On Mon, Feb 4, 2019 at 1:13 PM Vladimir Ozerov <vozerov@gridgain.com>
> > wrote:
> >
> > > Igor,
> > >
> > > Yes, cache groups are public API. However, we try to avoid new APIs
> > > depending on them.
> > > The main point from my side is that “similar cache group” can be easily
> > > generalized to “similar distribution”. This way we avoid cache groups
> on
> > > protocol level at virtually no cost.
> > >
> > > Vladimir.
> > >
> > > пн, 4 февр. 2019 г. в 12:48, Igor Sapego <isapego@apache.org>:
> > >
> > > > Guys,
> > > >
> > > > Can you explain why do we want to avoid Cache groups in protocol?
> > > >
> > > > If it's about simplicity of the protocol, then removing cache groups
> > will
> > > > not help much with it - we will still need to include "knownCacheIds"
> > > > field in request and "cachesWithTheSamePartitioning" field in
> response.
> > > > And also, since when do Ignite prefers simplicity over performance?
> > > >
> > > > If it's about not wanting to show internals of Ignite then it sounds
> > like
> > > > a very weak argument to me, since Cache Groups is a public thing [1].
> > > >
> > > > [1] - https://apacheignite.readme.io/docs/cache-groups
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > > >
> > > > On Mon, Feb 4, 2019 at 11:47 AM Vladimir Ozerov <
> vozerov@gridgain.com>
> > > > wrote:
> > > >
> > > > > Pavel, Igor,
> > > > >
> > > > > This is not very accurate to say that this will not save memory.
In
> > > > > practice we observed a number of OOME issues on the server-side due
> > to
> > > > many
> > > > > caches and it was one of motivations for cache groups (another one
> > disk
> > > > > access optimizations). On the other hand, I agree that we'd better
> to
> > > > avoid
> > > > > cache groups in the protocol because this is internal
> implementation
> > > > detail
> > > > > which is likely (I hope so) to be changed in future.
> > > > >
> > > > > So I have another proposal - let's track caches with the same
> > affinity
> > > > > distribution instead. That is, normally most of PARTITIONED caches
> > will
> > > > > have very few variants of configuration: it will be Rendezvous
> > affinity
> > > > > function, most likely with default partition number and with 1-2
> > > backups
> > > > at
> > > > > most. So when affinity distribution for specific cache is
> requested,
> > we
> > > > can
> > > > > append to the response *list of caches with the same distribution*.
> > > I.e.:
> > > > >
> > > > > class AffinityResponse {
> > > > >     Object distribution;    // Actual distribution
> > > > >     List<Integer> cacheIds; // Caches with similar distribution
> > > > > }
> > > > >
> > > > > Makes sense?
> > > > >
> > > > > On Sun, Feb 3, 2019 at 8:31 PM Pavel Tupitsyn <
> ptupitsyn@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Igor, I have a feeling that we should omit Cache Group stuff
from
> > the
> > > > > > protocol.
> > > > > > It is a rare use case and even then dealing with them on client
> > > barely
> > > > > > saves some memory.
> > > > > >
> > > > > > We can keep it simple and have partition map per cacheId.
> Thoughts?
> > > > > >
> > > > > > On Fri, Feb 1, 2019 at 6:49 PM Igor Sapego <isapego@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Guys, I've updated the proposal once again [1], so please,
> > > > > > > take a look and let me know what you think.
> > > > > > >
> > > > > > > [1] -
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Igor
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 17, 2019 at 1:05 PM Igor Sapego <
> isapego@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Yeah, I'll add it.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Igor
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 16, 2019 at 11:08 PM Pavel Tupitsyn <
> > > > > ptupitsyn@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> >  to every server
> > > > > > > >> I did not think of this issue. Now I agree with
your
> approach.
> > > > > > > >> Can you please add an explanation of this to the
IEP?
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> On Wed, Jan 16, 2019 at 2:53 PM Igor Sapego <
> > isapego@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Pavel,
> > > > > > > >> >
> > > > > > > >> > Yeah, it makes sense, but to me it seems
that this
> approach
> > > can
> > > > > lead
> > > > > > > >> > to more complicated client logic, as it will
require to
> make
> > > > > > > additional
> > > > > > > >> > call
> > > > > > > >> > to every server, that reports affinity topology
change.
> > > > > > > >> >
> > > > > > > >> > Guys, WDYT?
> > > > > > > >> >
> > > > > > > >> > Best Regards,
> > > > > > > >> > Igor
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Jan 15, 2019 at 10:59 PM Pavel Tupitsyn
<
> > > > > > ptupitsyn@apache.org
> > > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Igor,
> > > > > > > >> > >
> > > > > > > >> > > >  It is proposed to add flag to
every response, that
> > shows
> > > > > > whether
> > > > > > > >> the
> > > > > > > >> > > Affinity Topology Version of the cluster
has changed
> since
> > > the
> > > > > > last
> > > > > > > >> > request
> > > > > > > >> > > from the client.
> > > > > > > >> > > I propose to keep this flag. So no need
for periodic
> > checks.
> > > > > Makes
> > > > > > > >> sense?
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Jan 15, 2019 at 4:45 PM Igor
Sapego <
> > > > isapego@apache.org
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Pavel,
> > > > > > > >> > > >
> > > > > > > >> > > > This will require from client to
send this new request
> > > > > > > periodically,
> > > > > > > >> > I'm
> > > > > > > >> > > > not
> > > > > > > >> > > > sure this will make clients simpler.
Anyway, let's
> > discuss
> > > > it.
> > > > > > > >> > > >
> > > > > > > >> > > > Vladimir,
> > > > > > > >> > > >
> > > > > > > >> > > > With current proposal, we will
have affinity info in
> > > message
> > > > > > > header.
> > > > > > > >> > > >
> > > > > > > >> > > > Best Regards,
> > > > > > > >> > > > Igor
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Jan 15, 2019 at 11:01 AM
Vladimir Ozerov <
> > > > > > > >> vozerov@gridgain.com
> > > > > > > >> > >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Igor,
> > > > > > > >> > > > >
> > > > > > > >> > > > > I think that "Cache Partitions
Request" should
> contain
> > > > > > affinity
> > > > > > > >> > > topology
> > > > > > > >> > > > > version. Otherwise we do not
know what distribution
> is
> > > > > > returned
> > > > > > > -
> > > > > > > >> the
> > > > > > > >> > > one
> > > > > > > >> > > > > we expected, or some newer
one. The latter may
> happen
> > in
> > > > > case
> > > > > > > >> > topology
> > > > > > > >> > > > > changed or late affinity assignment
happened between
> > > > server
> > > > > > > >> response
> > > > > > > >> > > and
> > > > > > > >> > > > > subsequent client partitions
request.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Vladimir.
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Mon, Jan 14, 2019 at 6:08
PM Igor Sapego <
> > > > > > isapego@apache.org
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hello guys,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I've updated IEP page
[1] describing proposed
> > solution
> > > > in
> > > > > > more
> > > > > > > >> > > details
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > proposing some changes
for a protocol.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Please, take a look and
let me know what you
> think.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > [1] -
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Best Regards,
> > > > > > > >> > > > > > Igor
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Tue, Jun 19, 2018
at 11:54 AM Vladimir Ozerov <
> > > > > > > >> > > vozerov@gridgain.com
> > > > > > > >> > > > >
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Denis,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Yes, in principle
we can extend it. We are going
> > to
> > > > > > > implement
> > > > > > > >> it
> > > > > > > >> > in
> > > > > > > >> > > > > > > subsequent phases
of this IEP.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Tue, Jun 19,
2018 at 4:30 AM, Dmitriy
> > Setrakyan <
> > > > > > > >> > > > > > dsetrakyan@apache.org>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > On Mon, Jun
18, 2018 at 11:07 AM, Denis Magda
> <
> > > > > > > >> > dmagda@apache.org
> > > > > > > >> > > >
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Folks,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Feel that
this functionality can be extended
> > to
> > > > the
> > > > > > > >> automatic
> > > > > > > >> > > > > > > reconnect,
> > > > > > > >> > > > > > > > > can't
it? Presently we require to provide a
> > > static
> > > > > > list
> > > > > > > of
> > > > > > > >> > IPs
> > > > > > > >> > > to
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > > used
> > > > > > > >> > > > > > > > > at a reconnect
time. By having a partition
> map
> > > of
> > > > > all
> > > > > > > the
> > > > > > > >> > > nodes,
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > thin
> > > > > > > >> > > > > > > > > client
should be able to automate this
> piece.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Not sure if
static IP list can be avoided.
> What
> > > Igor
> > > > > is
> > > > > > > >> > > suggesting
> > > > > > > >> > > > is
> > > > > > > >> > > > > > > that
> > > > > > > >> > > > > > > > we try to pick
the best node out of the static
> > IP
> > > > > list.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > D.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message