ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Rudyak <irud...@gmail.com>
Subject Re: Memory leak in ignite-cassandra module
Date Tue, 12 Jun 2018 18:39:20 GMT
Denis,

I don't have ideas right now. First need to create a test to reproduce this
case. Then I'll have some ideas :-)

Igor

On Tue, Jun 12, 2018 at 11:26 AM, Denis Magda <dmagda@apache.org> wrote:

> Igor,
>
> Do you have any glues/ideas how to fix it? Is the provided information
> enough for you?
>
> --
> Denis
>
> On Mon, Jun 11, 2018 at 11:45 PM Igor Rudyak <irudyak@gmail.com> wrote:
>
> > Hi Kotamrajuyashasvi,
> >
> > Could you please create a ticket for this in Ignite JIRA? That's the
> > standard process to make improvements/fixes to Ignite.
> >
> > Thanks,
> > Igor Rudyak
> >
> > On Mon, Jun 11, 2018 at 11:36 PM, kotamrajuyashasvi <
> > kotamrajuyashasvi@gmail.com> wrote:
> >
> > > Hi
> > >
> > > We are working on an Ignite project with Cassandra as persistent
> storage.
> > > During our tests we faced the continuous cassandra session refresh
> issue.
> > > https://issues.apache.org/jira/browse/IGNITE-8354
> > >
> > > When we observed the above issue we also ran into OutOfMemory
> Exception.
> > > Though the above issue is solved we ran through the source code to find
> > out
> > > the root cause
> > > of OOM. We found one potential cause.
> > >
> > > In org.apache.ignite.cache.store.cassandra.session.
> > > CassandraSessionImpl.java
> > > when refresh() method is invoked to handle Exceptions, new Cluster is
> > build
> > > with same LoadBalancingPolicy Object. We are using RoundRobinPolicy so
> > same
> > > RoundRobinPolicy object would be used while building Cluster when
> > refresh()
> > > is invoked. In RoundRobinPolicy there is a CopyOnWriteArrayList<Host>
> > > liveHosts. When ever init(Cluster cluster, Collection<Host> hosts) is
> > > called
> > > on RoundRobinPolicy  it calls liveHosts.addAll(hosts) adding all the
> Host
> > > Object Collection to liveHosts.
> > > When ever Cluster is build during refresh() the Host Collection are
> added
> > > again to the liveHosts of the same RoundRobinPolicy that is used. Thus
> > same
> > > Hosts are added again to liveHosts for every refresh() and the size
> would
> > > grow indefinitely after many refresh() calls causing OOM. Even in the
> > heap
> > > dump post OOM we found huge number of Objects in liveHosts of
> > > RoundRobinPolicy Object.
> > >
> > > IGNITE-8354 has fixed the OOM by preventing unnecessary refresh() but
> > still
> > > does not fix the actual Memory leak caused due to RoundRobinPolicy .
> In a
> > > long run we can have many Cassandra refresh due to some genuine reasons
> > and
> > > then we end up with many Hosts in liveHosts of the RoundRobinPolicy
> > Object.
> > > Some possible solutions would be
> > > 1. To use new LoadBalancingPolicy object while building new Cluster
> > during
> > > refresh().
> > > 2. Somehow clear Objects in liveHosts during refresh().
> > >
> > > Also there's a work around to use DCAwareRoundRobinPolicy as it uses
> adds
> > > hosts dc wise and adds only if absent. But we are using single
> datacenter
> > > and its not recommended to use DCAwareRoundRobinPolicy when we have
> > single
> > > datacenter.
> > >
> > > I would like to request some one from ignite cassandra module
> development
> > > look into this issue.
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message