lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Bøgh Köster (JIRA) <j...@apache.org>
Subject [jira] [Updated] (SOLR-10506) Possible memory leak upon collection reload
Date Tue, 18 Apr 2017 10:17:41 GMT

     [ https://issues.apache.org/jira/browse/SOLR-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Torsten Bøgh Köster updated SOLR-10506:
---------------------------------------
    Attachment: solr_gc_path_via_zk_WatchManager.png

The strong references via schema index reader

> Possible memory leak upon collection reload
> -------------------------------------------
>
>                 Key: SOLR-10506
>                 URL: https://issues.apache.org/jira/browse/SOLR-10506
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: Server
>    Affects Versions: 6.5
>            Reporter: Torsten Bøgh Köster
>         Attachments: solr_collection_reload_13_cores.png, solr_gc_path_via_zk_WatchManager.png
>
>
> Upon manual Solr Collection reloading, references to the closed {{SolrCore}} are not
fully removed by the garbage collector as a strong reference to the {{ZkIndexSchemaReader}}
is held in a ZooKeeper {{Watcher}} that watches for schema changes.
> In our case, this leads to a massive memory leak as managed resources are still referenced
by the closed {{SolrCore}}. Our Solr cloud environment utilizes rather large managed resources
(synonyms, stopwords). To reproduce, we fired out environment up and reloaded the collection
13 times. As a result we fully exhausted our heap. A closer look with the Yourkit profiler
revealed 13 {{SolrCore}} instances, still holding strong references to the garbage collection
root (see screenshot 1).
> Each {{SolrCore}} instance holds a single path with strong references to the gc root
via a `Watcher` in `ZkIndexSchemaReader` (see screenshot 2). The {{ZkIndexSchemaReader}} registers
a close hook in the {{SolrCore}} but the Zookeeper is not removed upon core close.
> We supplied a Github Pull Request that extracts the zookeeper `Watcher` as a static inner
class. To eliminate the memory leak, the schema reader is held inside a `WeakReference` and
the reference is explicitly removed on core close.
> Initially I wanted to supply a test case but unfortunately did not find a good starting
point ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message