ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Memory leak in ignite-cassandra module
Date Tue, 12 Jun 2018 18:26:06 GMT
Igor,

Do you have any glues/ideas how to fix it? Is the provided information
enough for you?

--
Denis

On Mon, Jun 11, 2018 at 11:45 PM Igor Rudyak <irudyak@gmail.com> wrote:

> Hi Kotamrajuyashasvi,
>
> Could you please create a ticket for this in Ignite JIRA? That's the
> standard process to make improvements/fixes to Ignite.
>
> Thanks,
> Igor Rudyak
>
> On Mon, Jun 11, 2018 at 11:36 PM, kotamrajuyashasvi <
> kotamrajuyashasvi@gmail.com> wrote:
>
> > Hi
> >
> > We are working on an Ignite project with Cassandra as persistent storage.
> > During our tests we faced the continuous cassandra session refresh issue.
> > https://issues.apache.org/jira/browse/IGNITE-8354
> >
> > When we observed the above issue we also ran into OutOfMemory Exception.
> > Though the above issue is solved we ran through the source code to find
> out
> > the root cause
> > of OOM. We found one potential cause.
> >
> > In org.apache.ignite.cache.store.cassandra.session.
> > CassandraSessionImpl.java
> > when refresh() method is invoked to handle Exceptions, new Cluster is
> build
> > with same LoadBalancingPolicy Object. We are using RoundRobinPolicy so
> same
> > RoundRobinPolicy object would be used while building Cluster when
> refresh()
> > is invoked. In RoundRobinPolicy there is a CopyOnWriteArrayList<Host>
> > liveHosts. When ever init(Cluster cluster, Collection<Host> hosts) is
> > called
> > on RoundRobinPolicy  it calls liveHosts.addAll(hosts) adding all the Host
> > Object Collection to liveHosts.
> > When ever Cluster is build during refresh() the Host Collection are added
> > again to the liveHosts of the same RoundRobinPolicy that is used. Thus
> same
> > Hosts are added again to liveHosts for every refresh() and the size would
> > grow indefinitely after many refresh() calls causing OOM. Even in the
> heap
> > dump post OOM we found huge number of Objects in liveHosts of
> > RoundRobinPolicy Object.
> >
> > IGNITE-8354 has fixed the OOM by preventing unnecessary refresh() but
> still
> > does not fix the actual Memory leak caused due to RoundRobinPolicy . In a
> > long run we can have many Cassandra refresh due to some genuine reasons
> and
> > then we end up with many Hosts in liveHosts of the RoundRobinPolicy
> Object.
> > Some possible solutions would be
> > 1. To use new LoadBalancingPolicy object while building new Cluster
> during
> > refresh().
> > 2. Somehow clear Objects in liveHosts during refresh().
> >
> > Also there's a work around to use DCAwareRoundRobinPolicy as it uses adds
> > hosts dc wise and adds only if absent. But we are using single datacenter
> > and its not recommended to use DCAwareRoundRobinPolicy when we have
> single
> > datacenter.
> >
> > I would like to request some one from ignite cassandra module development
> > look into this issue.
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message