Return-Path: X-Original-To: apmail-usergrid-commits-archive@minotaur.apache.org Delivered-To: apmail-usergrid-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6F55117806 for ; Thu, 6 Nov 2014 00:14:48 +0000 (UTC) Received: (qmail 92787 invoked by uid 500); 6 Nov 2014 00:14:48 -0000 Delivered-To: apmail-usergrid-commits-archive@usergrid.apache.org Received: (qmail 92765 invoked by uid 500); 6 Nov 2014 00:14:48 -0000 Mailing-List: contact commits-help@usergrid.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@usergrid.incubator.apache.org Delivered-To: mailing list commits@usergrid.incubator.apache.org Received: (qmail 92641 invoked by uid 99); 6 Nov 2014 00:14:48 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Nov 2014 00:14:48 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id BF43B909BC7; Thu, 6 Nov 2014 00:14:47 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: toddnine@apache.org To: commits@usergrid.apache.org Date: Thu, 06 Nov 2014 00:14:55 -0000 Message-Id: <4ac5fdf8aebb49ae8e572f31627ac1bc@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [09/50] git commit: Fixes sysconfigs for elasticsearch, as well as adds sensible recovery defaults so we don't saturate the system during recovery Fixes sysconfigs for elasticsearch, as well as adds sensible recovery defaults so we don't saturate the system during recovery Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/f85b562c Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/f85b562c Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/f85b562c Branch: refs/heads/two-dot-o Commit: f85b562c7394da7ba3cc9073d88bbc9a76a660b6 Parents: 5a5377b Author: Todd Nine Authored: Fri Oct 31 15:44:44 2014 -0600 Committer: Todd Nine Committed: Fri Oct 31 15:44:44 2014 -0600 ---------------------------------------------------------------------- .../dist/init_instance/install_elasticsearch.sh | 18 ++++++++++++ .../main/groovy/configure_elasticsearch.groovy | 31 ++++++++++++++++++-- 2 files changed, 47 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/f85b562c/stack/awscluster/src/main/dist/init_instance/install_elasticsearch.sh ---------------------------------------------------------------------- diff --git a/stack/awscluster/src/main/dist/init_instance/install_elasticsearch.sh b/stack/awscluster/src/main/dist/init_instance/install_elasticsearch.sh index 06ca60b..dfbe929 100644 --- a/stack/awscluster/src/main/dist/init_instance/install_elasticsearch.sh +++ b/stack/awscluster/src/main/dist/init_instance/install_elasticsearch.sh @@ -35,6 +35,13 @@ chown elasticsearch /mnt/data/elasticsearch mkdir -p /mnt/log/elasticsearch chown elasticsearch /mnt/log/elasticsearch +mkdir -p /mnt/var/log/elasticsearch +chown elasticsearch /mnt/var/log/elasticsearch + +#Set up the symlink for the logs +rm -rf /var/log/elasticsearch +ln -s /mnt/var/log/elasticsearch /var/log/elasticsearch + # Configure ElasticSearch cd /usr/share/usergrid/scripts @@ -84,13 +91,24 @@ case `(curl http://169.254.169.254/latest/meta-data/instance-type)` in export ES_HEAP_SIZE=24g esac + + + cat >> /etc/default/elasticsearch << EOF ES_HEAP_SIZE=${ES_HEAP_SIZE} MAX_OPEN_FILES=65535 MAX_LOCKED_MEMORY=unlimited JAVA_HOME=/usr/lib/jvm/jdk1.7.0 +LOG_DIR=/var/log/elasticsearch EOF + +cat >> /etc/security/limits.conf << EOF +elasticsearch - nofile 65535 +elasticsearch - memlock unlimited +EOF + + groovy ./configure_elasticsearch.groovy > /etc/elasticsearch/elasticsearch.yml update-rc.d elasticsearch defaults 95 10 http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/f85b562c/stack/awscluster/src/main/groovy/configure_elasticsearch.groovy ---------------------------------------------------------------------- diff --git a/stack/awscluster/src/main/groovy/configure_elasticsearch.groovy b/stack/awscluster/src/main/groovy/configure_elasticsearch.groovy index 0a22d27..5e853f0 100644 --- a/stack/awscluster/src/main/groovy/configure_elasticsearch.groovy +++ b/stack/awscluster/src/main/groovy/configure_elasticsearch.groovy @@ -58,20 +58,47 @@ network: path: logs: /mnt/log/elasticsearch data: /mnt/data/elasticsearch + bootstrap.mlockall: true threadpool.index.type: fixed threadpool.index.size: 160 threadpool.index.queue_size: 401 threadpool.bulk.type: fixed -threadpool.bulk.size: 500 +threadpool.bulk.size: 10000 threadpool.bulk.queue_size: 800 -threadpool.search.size: 1000 +threadpool.search.size: 20000 threadpool.search.type: fixed threadpool.search.queue_size: 1000 action.auto_create_index: false action.disable_delete_all_indices: true + +################################# +# Operational settings taken from a loggly blog here. Tweak and work as required +# https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/ +################################# + +#Only cache 25% of our available memory +indices.fielddata.cache.size: 25% + +#If you haven't used it in 10 minutes, evict it from the cache +indices.fielddata.cache.expire: 10m + +#Only allow rebalancing of 2 shards at a time +cluster.routing.allocation.cluster_concurrent_rebalance:2 + +#Re-shard when our disks start getting full +cluster.routing.allocation.disk.threshold_enabled:true +cluster.routing.allocation.disk.watermark.low:.97 +cluster.routing.allocation.disk.watermark.high:.99 + +#Set streaming high water marks so reboots don't kill our service +cluster.routing.allocation.node_concurrent_recoveries:4 +cluster.routing.allocation.node_initial_primaries_recoveries:18 +indices.recovery.concurrent_streams: 4 +indices.recovery.max_bytes_per_sec: 40mb + """ println elasticSearchConfig