lucene-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From er...@apache.org
Subject [lucene-solr] branch master updated: SOLR-13771: Add -v and -m to ulimit section of reference guide and bin/solr checks
Date Mon, 30 Sep 2019 21:58:48 GMT
This is an automated email from the ASF dual-hosted git repository.

erick pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git


The following commit(s) were added to refs/heads/master by this push:
     new a1f3d2c  SOLR-13771: Add -v and -m to ulimit section of reference guide and bin/solr
checks
a1f3d2c is described below

commit a1f3d2c29a1b61ac01e5defcb097695c43aaadd9
Author: Erick Erickson <Erick.Erickson@gmail.com>
AuthorDate: Mon Sep 30 17:58:35 2019 -0400

    SOLR-13771: Add -v and -m to ulimit section of reference guide and bin/solr checks
---
 solr/bin/solr                                      | 19 ++++++++++++++++--
 .../src/taking-solr-to-production.adoc             | 23 ++++++++++++++--------
 2 files changed, 32 insertions(+), 10 deletions(-)

diff --git a/solr/bin/solr b/solr/bin/solr
index 55cb147..596242f 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -1521,6 +1521,8 @@ if [ -z "$SOLR_ULIMIT_CHECKS" ] || [ "$SOLR_ULIMIT_CHECKS" != "false"
]; then
     if hash ulimit 2>/dev/null; then
        openFiles=$(ulimit -n)
        maxProcs=$(ulimit -u)
+       virtualMemory=$(ulimit -v)
+       maxMemory=$(ulimit -m)
        if [ $openFiles != "unlimited" ] && [ $openFiles -lt "$SOLR_RECOMMENDED_OPEN_FILES"
]; then
            echo "*** [WARN] *** Your open file limit is currently $openFiles.  "
            echo " It should be set to $SOLR_RECOMMENDED_OPEN_FILES to avoid operational disruption.
"
@@ -1532,10 +1534,23 @@ if [ -z "$SOLR_ULIMIT_CHECKS" ] || [ "$SOLR_ULIMIT_CHECKS" != "false"
]; then
            echo " It should be set to $SOLR_RECOMMENDED_MAX_PROCESSES to avoid operational
disruption. "
            echo " If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false
in your profile or solr.in.sh"
        fi
+       if [ $virtualMemory != "unlimited" ]; then
+           echo "*** [WARN] ***  Your Virtual Memory limit is $virtualMemory. "
+           echo " It should be set to 'unlimited' to avoid operational disruption. "
+           echo " If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false
in your profile or solr.in.sh"
+       fi
+       if [ $maxMemory != "unlimited" ]; then
+           echo "*** [WARN] ***  Your Max Memory Size limit is $maxMemory. "
+           echo " It should be set to 'unlimited' to avoid operational disruption. "
+           echo " If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false
in your profile or solr.in.sh"
+       fi
+
     else
       echo "Could not check ulimits for processes and open files, recommended values are"
-      echo "     max processes: $SOLR_RECOMMENDED_MAX_PROCESSES "
-      echo "     open files:    $SOLR_RECOMMENDED_OPEN_FILES"
+      echo "     max processes:   $SOLR_RECOMMENDED_MAX_PROCESSES "
+      echo "     open files:      $SOLR_RECOMMENDED_OPEN_FILES"
+      echo "     virtual memory:  unlimited"
+      echo "     max memorh size: unlimited"
     fi
   fi
 fi
diff --git a/solr/solr-ref-guide/src/taking-solr-to-production.adoc b/solr/solr-ref-guide/src/taking-solr-to-production.adoc
index 9e3595d..954c4dd 100644
--- a/solr/solr-ref-guide/src/taking-solr-to-production.adoc
+++ b/solr/solr-ref-guide/src/taking-solr-to-production.adoc
@@ -273,26 +273,33 @@ The `bin/solr` script simply passes options starting with `-D` on to
the JVM dur
 SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=10000"
 ----
 
-=== File Handles and Processes (ulimit settings)
+=== Ulimit settings (*nix operating systems)
 
-Two common settings that result in errors on *nix systems are file handles and user processes.
-
-It is common for the default limits for number of processes and file handles to default to
values that are too low for a large Solr installation. The required number of each of these
will increase based on a combination of the number of replicas hosted per node and the number
of segments in the index for each replica.
-
-The usual recommendation is to make processes and file handles at least 65,000 each, unlimited
if possible. On most *nix systems, this command will show the currently-defined limits:
+There are several settings that should be monitored and set as high as possible, "unlimited"
by preference. On most "*nix" operating systems, you can see the current values by typing
the following at a command prompt.
 
 [source,bash]
 ----
 ulimit -a
 ----
 
-It is strongly recommended that file handle and process limits be permanently raised as above.
The exact form of the command will vary per operating system, and some systems require editing
configuration files and restarting your server. Consult your system administrators for guidance
in your particular environment.
+These four settings in particular are important to have set very high, unlimited by preference.
+
+ * max processes (ulimit -u): 65,000 is the recommended _minimum_
+ * file handles (ulimit -n): 65,000 is the recommended _minimum_. All the files used by all
replicas have their file handles open at once so this can grow quite large.
+ * virtual memory (ulimit -v): Set to unlimited. This is used to by MMapping the indexes.
+ * max memory size (ulimit -m): Also used by MMap, set to unlimited.
+ * If your system supports it, `sysctl vm.max_map_count`, should be set to unlimited as well.
+
+We strongly recommend that these settings be permanently raised. The exact process to permanently
raise them will vary per operating system. Some systems require editing configuration files
and restarting your server. Consult your system administrators for guidance in your particular
environment.
+[WARNING]
+====
+Check these limits every time you upgrade your kernel or operating system. These operations
can reset these to their defaults.
+====
 
 [WARNING]
 ====
 If these limits are exceeded, the problems reported by Solr vary depending on the specific
operation responsible for exceeding the limit. Errors such as "too many open files", "connection
error", and "max processes exceeded" have been reported, as well as SolrCloud recovery failures.
 
-Since exceeding these limits can result in such varied symptoms it is _strongly_ recommended
that these limits be permanently raised as recommended above.
 ====
 
 == Running Multiple Solr Nodes per Host


Mime
View raw message