cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Dlugi (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
Date Thu, 31 Aug 2017 07:39:00 GMT


Markus Dlugi commented on CASSANDRA-13754:

[~jjirsa], we are experiencing the same issue on Cassandra 3.11.0 and Netty 4.0.44. We have
some custom triggers which create additional rows when {{INSERT}}s on specific CFs are executed.
When inserting a lot of data, the Cassandra would constantly crash with {{OutOfMemoryError}}s.
We analyzed a heap dump and came to the same conclusion, which is that the instances of {{FastThreadLocalThread}}
were responsible for this behaviour. As a quick workaround, we added a call to {{FastThreadLocal.removeAll()}}
to our triggers, which alleviates the problem for us.

It seems that there already was an issue once with {{FastThreadLocal}}s, as described in CASSANDRA-13033,
but apparently the change by [~snazy] in CASSANDRA-13034 resurfaced the issue. For a proper
fix, I think all {{FastThreadLocalThread}} instances should call {{FastThreadLocal.removeAll()}}
when they are done with their work. A quick glance at the code indicates that this affects
the classes {{CompressedInputStream}}, {{StreamingInboundHandler}}, {{NamedThreadFactory}}
and {{SEPWorker}}.

> FastThreadLocal leaks memory
> ----------------------------
>                 Key: CASSANDRA-13754
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>            Reporter: Eric Evans
>             Fix For: 3.11.1
> After a chronic bout of {{OutOfMemoryError}} in our development environment, a heap analysis
is showing that more than 10G of our 12G heaps are consumed by the {{threadLocals}} members
(instances of {{java.lang.ThreadLocalMap}}) of various {{io.netty.util.concurrent.FastThreadLocalThread}}
instances.  Reverting [cecbe17|;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
fixes the issue.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message