lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-3685) solrcloud crashes on startup due to excessive memory consumption
Date Fri, 03 Aug 2012 17:44:03 GMT

    [ https://issues.apache.org/jira/browse/SOLR-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428255#comment-13428255
] 

Mark Miller commented on SOLR-3685:
-----------------------------------

Is it 2 or 3 cores you have? One thing is that it won't be just one extra searcher and index
- it will be that times the number of cores. All of them will attempt to recover at the same
time. So you will see a bump in RAM reqs. You are talking about off heap RAM though - I don't
think SolrCloud will have much to do with that.

Looking at your logs, it appears that you are replicating because the transaction logs look
suspect - probably because of a hard power down. If you shutdown gracefully, you would get
a peer sync instead which should determine you are up to date.

The comment for the path you are taking says:

{quote}
        // last operation at the time of startup had the GAP flag set...
        // this means we were previously doing a full index replication
        // that probably didn't complete and buffering updates in the meantime.
{quote}
                
> solrcloud crashes on startup due to excessive memory consumption
> ----------------------------------------------------------------
>
>                 Key: SOLR-3685
>                 URL: https://issues.apache.org/jira/browse/SOLR-3685
>             Project: Solr
>          Issue Type: Bug
>          Components: replication (java), SolrCloud
>    Affects Versions: 4.0-ALPHA
>         Environment: Debian GNU/Linux Squeeze 64bit
> Solr 5.0-SNAPSHOT 1365667M - markus - 2012-07-25 19:09:43
>            Reporter: Markus Jelsma
>            Priority: Critical
>             Fix For: 4.1
>
>         Attachments: info.log
>
>
> There's a serious problem with restarting nodes, not cleaning old or unused index directories
and sudden replication and Java being killed by the OS due to excessive memory allocation.
Since SOLR-1781 was fixed index directories get cleaned up when a node is being restarted
cleanly, however, old or unused index directories still pile up if Solr crashes or is being
killed by the OS, happening here.
> We have a six-node 64-bit Linux test cluster with each node having two shards. There's
512MB RAM available and no swap. Each index is roughly 27MB so about 50MB per node, this fits
easily and works fine. However, if a node is being restarted, Solr will consistently crash
because it immediately eats up all RAM. If swap is enabled Solr will eat an additional few
100MB's right after start up.
> This cannot be solved by restarting Solr, it will just crash again and leave index directories
in place until the disk is full. The only way i can restart a node safely is to delete the
index directories and have it replicate from another node. If i then restart the node it will
crash almost consistently.
> I'll attach a log of one of the nodes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message