From solr-user-return-143464-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Mon Sep 3 19:33:17 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1A418180647 for ; Mon, 3 Sep 2018 19:33:16 +0200 (CEST) Received: (qmail 11225 invoked by uid 500); 3 Sep 2018 17:33:14 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 10207 invoked by uid 99); 3 Sep 2018 17:33:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Sep 2018 17:33:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 92466180A49 for ; Mon, 3 Sep 2018 17:33:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.474 X-Spam-Level: ** X-Spam-Status: No, score=2.474 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FROM_EXCESS_BASE64=0.105, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id SPDC3ZPb0hph for ; Mon, 3 Sep 2018 17:33:12 +0000 (UTC) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 32CA05F174 for ; Mon, 3 Sep 2018 17:33:12 +0000 (UTC) Received: by mail-wm0-f67.google.com with SMTP id c14-v6so1890246wmb.4 for ; Mon, 03 Sep 2018 10:33:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:mime-version:subject:message-id:date:to; bh=2KSnFoatvMmOaORpDBJiY+OVWeL0TDCtfRgCWKjy0bg=; b=Kj3o2YBnSNDAoOl5yQqBUAJQmV7Z+7ctZFw73btDlizRd885nviCMwQMcdZTKVG15b yzRxh/RzfJ1LwaPifVpRYw2EhrajOH8PX+lrUYN36GreKAEmfV+pLw66Xusyj9jmn13F elOUXkEmgogNjKuqlgZ08HmGnhQOA4vd/B+0ZpFMA0A6KP4ysEVEBplWl/9pqUzLIK2D LdD7VYBGJHL1QjzSqv1Qs3iGqBW+yw3jslk7ty74SEpcCaczbOuRJFs2pzJRCwUr0OOj Mu/KK0c2YF3KuHGHCkworzwBeKnmQ14IcJOpOU7DaN++a6yRC78zlUiYsdyjyiCLOMyb +9Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:message-id:date:to; bh=2KSnFoatvMmOaORpDBJiY+OVWeL0TDCtfRgCWKjy0bg=; b=dC4insUXz5sfsQMrY9X/2kZw9Trg6W+PPI03zfgrQXtbq4t6kd4DdlP2QHtl+QlluZ TA7VKUiyzlOPJjIu1wsCbGHsX83TRbZWBRhQPIpsOeGFYwsrPkASRdqTjk7n99QWIysj ygTCe462MLkNLEqN5dAKXqQINQis/sr9Bwk7hsuL6wrre2I2qQEA69I+rOUzL6sz9byr 4CHwPvVatAABOz1EEI/aRt1VWQ8VZZOtmLHTjxVlk4Bb+5r6POPYzL0QdiNmv/tHFOSQ aMJQnVU6PPqnCf2U5NTViw9nmQ0Pxe7+YX6+xmu/SeByyBgLq/si7JMrcy4RtRK3s/iz XD5A== X-Gm-Message-State: APzg51DMNXBMAOBwv2yahECeTtijgLo6SUkUduYLb/UwEOspdUgKDRHz 5GVwZSGFRLMa/0XVLtx9OPrEMwGF X-Google-Smtp-Source: ANB0VdZ+jgH8agIcBaaXRV5Vsx7P7v7iYACXCCC7fQNz5BDzEDOTsVXMEUCdufbcMbGRc2k67ojnjA== X-Received: by 2002:a1c:2ed4:: with SMTP id u203-v6mr296833wmu.19.1535995990976; Mon, 03 Sep 2018 10:33:10 -0700 (PDT) Received: from [172.17.0.93] ([89.27.166.58]) by smtp.gmail.com with ESMTPSA id 94-v6sm17141864wrc.10.2018.09.03.10.33.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Sep 2018 10:33:10 -0700 (PDT) From: =?utf-8?B?QmrDtnJuIEjDpHVzZXI=?= Content-Type: multipart/alternative; boundary="Apple-Mail=_C58ADA7B-2A8D-43A6-BDDB-3913FA24DE30" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Heap Memory Problem after Upgrading to 7.4.0 Message-Id: <7B5E78C6-8CF6-42EE-8D28-872230DEDCFB@gmail.com> Date: Mon, 3 Sep 2018 19:33:09 +0200 To: solr-user@lucene.apache.org X-Mailer: Apple Mail (2.3445.9.1) --Apple-Mail=_C58ADA7B-2A8D-43A6-BDDB-3913FA24DE30 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello, we recently upgraded our solrcloud (5 nodes, 25 collections, 1 shard = each, 4 replicas each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. = We are running Zookeeper 4.1.13. Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap space = exhaustion. After obtaining a heap dump it looks like that we have a lot = of IndexSearchers open for our largest collection. The dump contains around ~60 IndexSearchers, and each containing around = ~40mb heap. Another 500MB of heap is the fieldcache, which is expected = in my opinion. The current config can be found here: = https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844 = Analyzing the heap dump eclipse MAT says this: Problem Suspect 1 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded by = "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy = 1.981.148.336 (38,26%) bytes.=20 Biggest instances: =E2=80=A2 org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 = - 70.087.272 (1,35%) bytes.=20 =E2=80=A2 org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 = - 65.678.264 (1,27%) bytes.=20 =E2=80=A2 org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 = - 63.050.600 (1,22%) bytes.=20 Problem Suspect 2 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded by = "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy = 1.373.110.208 (26,52%) bytes.=20 Any help is appreciated. Thank you very much! Bj=C3=B6rn= --Apple-Mail=_C58ADA7B-2A8D-43A6-BDDB-3913FA24DE30--