Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B2637200CD0 for ; Tue, 11 Jul 2017 01:09:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B0CCF16494A; Mon, 10 Jul 2017 23:09:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CFB83164948 for ; Tue, 11 Jul 2017 01:09:11 +0200 (CEST) Received: (qmail 15276 invoked by uid 500); 10 Jul 2017 23:09:11 -0000 Mailing-List: contact commits-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.incubator.apache.org Delivered-To: mailing list commits@impala.incubator.apache.org Received: (qmail 15267 invoked by uid 99); 10 Jul 2017 23:09:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jul 2017 23:09:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8C1C7194655 for ; Mon, 10 Jul 2017 23:09:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.222 X-Spam-Level: X-Spam-Status: No, score=-4.222 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Ts89TxteOZgC for ; Mon, 10 Jul 2017 23:09:09 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id DC9445FDE4 for ; Mon, 10 Jul 2017 23:09:08 +0000 (UTC) Received: (qmail 15262 invoked by uid 99); 10 Jul 2017 23:09:08 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jul 2017 23:09:08 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 9C708DF9C3; Mon, 10 Jul 2017 23:09:07 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: jrussell@apache.org To: commits@impala.incubator.apache.org Date: Mon, 10 Jul 2017 23:09:08 -0000 Message-Id: <61f655062123415ca9a044eb395712a5@git.apache.org> In-Reply-To: <1f1a8538e4c34e52a2d57fe21914c1b0@git.apache.org> References: <1f1a8538e4c34e52a2d57fe21914c1b0@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [2/3] incubator-impala git commit: [DOCS] Advise setting vm.overcommit_memory=1 in various places archived-at: Mon, 10 Jul 2017 23:09:12 -0000 [DOCS] Advise setting vm.overcommit_memory=1 in various places Reusing the same advice under "Known Issues", scalability considerations, and in the Impala + Kerberos section. Change-Id: Icbfa755e2c9769a8458fd93362769856cf32e301 Reviewed-on: http://gerrit.cloudera.org:8080/7349 Reviewed-by: Mostafa Mokhtar Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/db3f3235 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/db3f3235 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/db3f3235 Branch: refs/heads/master Commit: db3f3235d7c28ff9963e0332b830ecc86f54a505 Parents: cd44381 Author: John Russell Authored: Fri Jun 30 16:00:30 2017 -0700 Committer: Impala Public Jenkins Committed: Mon Jul 10 23:06:45 2017 +0000 ---------------------------------------------------------------------- docs/shared/impala_common.xml | 32 ++++++++++++++++++++++++++++++++ docs/topics/impala_kerberos.xml | 8 ++++++++ docs/topics/impala_known_issues.xml | 11 +++++++++++ docs/topics/impala_scalability.xml | 8 ++++++++ 4 files changed, 59 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/shared/impala_common.xml ---------------------------------------------------------------------- diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml index 6e65c40..2ec2537 100644 --- a/docs/shared/impala_common.xml +++ b/docs/shared/impala_common.xml @@ -1365,6 +1365,38 @@ select c_first_name, c_last_name from customer where lower(trim(c_last_name)) rl user@example.com, in a Kerberized environment.

+

+ On a kerberized cluster with high memory utilization, kinit commands executed after + every 'kerberos_reinit_interval' may cause out-of-memory errors, because executing + the command involves a fork of the Impala process. The error looks similar to the following: +principal_details +Failed to execute shell cmd: 'kinit -k -t keytab_details', +error was: Error(12): Cannot allocate memory +]]> + +

+ +

+ The following command changes the vm.overcommit_memory + setting immediately on a running host. However, this setting is reset + when the host is restarted. + /proc/sys/vm/overcommit_memory +]]> + +

+

+ To change the setting in a persistent way, add the following line to the + /etc/sysctl.conf file: + + +

+

+ Then run sysctl -p. No reboot is needed. +

  • Currently, each Impala GRANT or REVOKE statement can only grant or http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_kerberos.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_kerberos.xml b/docs/topics/impala_kerberos.xml index 5032fba..5d97aeb 100644 --- a/docs/topics/impala_kerberos.xml +++ b/docs/topics/impala_kerberos.xml @@ -351,4 +351,12 @@ $ chown impala:impala impala-http.keytab + + Kerberos-Related Memory Overhead for Large Clusters + +

    +

    + + + http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_known_issues.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_known_issues.xml b/docs/topics/impala_known_issues.xml index 30cc245..51920c9 100644 --- a/docs/topics/impala_known_issues.xml +++ b/docs/topics/impala_known_issues.xml @@ -623,6 +623,17 @@ impala hard nproc 262144 + + Kerberos initialization errors due to high memory usage + +

    +

    Bug: IMPALA-2294

    +

    Severity: High

    +

    Workaround:

    +

    + + + DROP TABLE PURGE on S3A table may not delete externally written files http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_scalability.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_scalability.xml b/docs/topics/impala_scalability.xml index af61e7a..2ea7603 100644 --- a/docs/topics/impala_scalability.xml +++ b/docs/topics/impala_scalability.xml @@ -887,6 +887,14 @@ so other secure services might be affected temporarily. + + Kerberos-Related Memory Overhead for Large Clusters + +

    +

    + + + Avoiding CPU Hotspots for HDFS Cached Data