ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Rudyak <irud...@gmail.com>
Subject Re: Memory leak in ignite-cassandra module
Date Tue, 12 Jun 2018 18:40:28 GMT
I will be also good to know which version of Cassandra driver was used to
run into OOM exception.

Igor

On Tue, Jun 12, 2018 at 11:39 AM, Igor Rudyak <irudyak@gmail.com> wrote:

> Denis,
>
> I don't have ideas right now. First need to create a test to reproduce
> this case. Then I'll have some ideas :-)
>
> Igor
>
> On Tue, Jun 12, 2018 at 11:26 AM, Denis Magda <dmagda@apache.org> wrote:
>
>> Igor,
>>
>> Do you have any glues/ideas how to fix it? Is the provided information
>> enough for you?
>>
>> --
>> Denis
>>
>> On Mon, Jun 11, 2018 at 11:45 PM Igor Rudyak <irudyak@gmail.com> wrote:
>>
>> > Hi Kotamrajuyashasvi,
>> >
>> > Could you please create a ticket for this in Ignite JIRA? That's the
>> > standard process to make improvements/fixes to Ignite.
>> >
>> > Thanks,
>> > Igor Rudyak
>> >
>> > On Mon, Jun 11, 2018 at 11:36 PM, kotamrajuyashasvi <
>> > kotamrajuyashasvi@gmail.com> wrote:
>> >
>> > > Hi
>> > >
>> > > We are working on an Ignite project with Cassandra as persistent
>> storage.
>> > > During our tests we faced the continuous cassandra session refresh
>> issue.
>> > > https://issues.apache.org/jira/browse/IGNITE-8354
>> > >
>> > > When we observed the above issue we also ran into OutOfMemory
>> Exception.
>> > > Though the above issue is solved we ran through the source code to
>> find
>> > out
>> > > the root cause
>> > > of OOM. We found one potential cause.
>> > >
>> > > In org.apache.ignite.cache.store.cassandra.session.
>> > > CassandraSessionImpl.java
>> > > when refresh() method is invoked to handle Exceptions, new Cluster is
>> > build
>> > > with same LoadBalancingPolicy Object. We are using RoundRobinPolicy so
>> > same
>> > > RoundRobinPolicy object would be used while building Cluster when
>> > refresh()
>> > > is invoked. In RoundRobinPolicy there is a CopyOnWriteArrayList<Host>
>> > > liveHosts. When ever init(Cluster cluster, Collection<Host> hosts)
is
>> > > called
>> > > on RoundRobinPolicy  it calls liveHosts.addAll(hosts) adding all the
>> Host
>> > > Object Collection to liveHosts.
>> > > When ever Cluster is build during refresh() the Host Collection are
>> added
>> > > again to the liveHosts of the same RoundRobinPolicy that is used. Thus
>> > same
>> > > Hosts are added again to liveHosts for every refresh() and the size
>> would
>> > > grow indefinitely after many refresh() calls causing OOM. Even in the
>> > heap
>> > > dump post OOM we found huge number of Objects in liveHosts of
>> > > RoundRobinPolicy Object.
>> > >
>> > > IGNITE-8354 has fixed the OOM by preventing unnecessary refresh() but
>> > still
>> > > does not fix the actual Memory leak caused due to RoundRobinPolicy .
>> In a
>> > > long run we can have many Cassandra refresh due to some genuine
>> reasons
>> > and
>> > > then we end up with many Hosts in liveHosts of the RoundRobinPolicy
>> > Object.
>> > > Some possible solutions would be
>> > > 1. To use new LoadBalancingPolicy object while building new Cluster
>> > during
>> > > refresh().
>> > > 2. Somehow clear Objects in liveHosts during refresh().
>> > >
>> > > Also there's a work around to use DCAwareRoundRobinPolicy as it uses
>> adds
>> > > hosts dc wise and adds only if absent. But we are using single
>> datacenter
>> > > and its not recommended to use DCAwareRoundRobinPolicy when we have
>> > single
>> > > datacenter.
>> > >
>> > > I would like to request some one from ignite cassandra module
>> development
>> > > look into this issue.
>> > >
>> > >
>> > >
>> > > --
>> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message