impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jruss...@apache.org
Subject [2/3] incubator-impala git commit: [DOCS] Advise setting vm.overcommit_memory=1 in various places
Date Mon, 10 Jul 2017 23:09:08 GMT
[DOCS] Advise setting vm.overcommit_memory=1 in various places

Reusing the same advice under "Known Issues", scalability
considerations, and in the Impala + Kerberos section.

Change-Id: Icbfa755e2c9769a8458fd93362769856cf32e301
Reviewed-on: http://gerrit.cloudera.org:8080/7349
Reviewed-by: Mostafa Mokhtar <mmokhtar@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/db3f3235
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/db3f3235
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/db3f3235

Branch: refs/heads/master
Commit: db3f3235d7c28ff9963e0332b830ecc86f54a505
Parents: cd44381
Author: John Russell <jrussell@cloudera.com>
Authored: Fri Jun 30 16:00:30 2017 -0700
Committer: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
Committed: Mon Jul 10 23:06:45 2017 +0000

----------------------------------------------------------------------
 docs/shared/impala_common.xml       | 32 ++++++++++++++++++++++++++++++++
 docs/topics/impala_kerberos.xml     |  8 ++++++++
 docs/topics/impala_known_issues.xml | 11 +++++++++++
 docs/topics/impala_scalability.xml  |  8 ++++++++
 4 files changed, 59 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 6e65c40..2ec2537 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -1365,6 +1365,38 @@ select c_first_name, c_last_name from customer where lower(trim(c_last_name))
rl
         <codeph>user@example.com</codeph>, in a Kerberized environment.
       </p>
 
+      <p id="vm_overcommit_memory_intro">
+        On a kerberized cluster with high memory utilization, <cmdname>kinit</cmdname>
commands executed after
+        every <codeph>'kerberos_reinit_interval'</codeph> may cause out-of-memory
errors, because executing
+        the command involves a fork of the Impala process. The error looks similar to the
following:
+<codeblock><![CDATA[
+Failed to obtain Kerberos ticket for principal: <varname>principal_details</varname>
+Failed to execute shell cmd: 'kinit -k -t <varname>keytab_details</varname>',
+error was: Error(12): Cannot allocate memory
+]]>
+</codeblock>
+      </p>
+
+      <p id="vm_overcommit_memory_start">
+        The following command changes the <codeph>vm.overcommit_memory</codeph>
+        setting immediately on a running host. However, this setting is reset
+        when the host is restarted.
+<codeblock><![CDATA[
+echo 1 > /proc/sys/vm/overcommit_memory
+]]>
+</codeblock>
+      </p>
+      <p>
+        To change the setting in a persistent way, add the following line to the
+        <filepath>/etc/sysctl.conf</filepath> file:
+<codeblock><![CDATA[
+vm.overcommit_memory=1
+]]>
+</codeblock>
+      </p>
+      <p id="vm_overcommit_memory_end">
+        Then run <codeph>sysctl -p</codeph>. No reboot is needed.
+      </p>
       <ul>
         <li id="grant_revoke_single">
           Currently, each Impala <codeph>GRANT</codeph> or <codeph>REVOKE</codeph>
statement can only grant or

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_kerberos.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_kerberos.xml b/docs/topics/impala_kerberos.xml
index 5032fba..5d97aeb 100644
--- a/docs/topics/impala_kerberos.xml
+++ b/docs/topics/impala_kerberos.xml
@@ -351,4 +351,12 @@ $ chown impala:impala impala-http.keytab</codeblock>
     </conbody>
   </concept>
 
+  <concept rev="IMPALA-2294" id="kerberos_overhead_memory_usage">
+  <title>Kerberos-Related Memory Overhead for Large Clusters</title>
+  <conbody>
+    <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_intro"/>
+    <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_start" conrefend="vm_overcommit_memory_end"/>
+  </conbody>
+  </concept>
+
 </concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_known_issues.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_known_issues.xml b/docs/topics/impala_known_issues.xml
index 30cc245..51920c9 100644
--- a/docs/topics/impala_known_issues.xml
+++ b/docs/topics/impala_known_issues.xml
@@ -623,6 +623,17 @@ impala hard nproc 262144
       </conbody>
     </concept>
 
+    <concept id="IMPALA-2294">
+      <title>Kerberos initialization errors due to high memory usage</title>
+      <conbody>
+        <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_intro"/>
+        <p><b>Bug:</b> <xref keyref="IMPALA-2294">IMPALA-2294</xref></p>
+        <p><b>Severity:</b> High</p>
+        <p><b>Workaround:</b></p>
+        <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_start" conrefend="vm_overcommit_memory_end"/>
+      </conbody>
+    </concept>
+
     <concept id="drop_table_purge_s3a">
       <title>DROP TABLE PURGE on S3A table may not delete externally written files</title>
       <conbody>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_scalability.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_scalability.xml b/docs/topics/impala_scalability.xml
index af61e7a..2ea7603 100644
--- a/docs/topics/impala_scalability.xml
+++ b/docs/topics/impala_scalability.xml
@@ -887,6 +887,14 @@ so other secure services might be affected temporarily.
 </conbody>
 </concept>
 
+  <concept rev="IMPALA-2294" id="kerberos_overhead_memory_usage">
+  <title>Kerberos-Related Memory Overhead for Large Clusters</title>
+  <conbody>
+    <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_intro"/>
+    <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_start" conrefend="vm_overcommit_memory_end"/>
+  </conbody>
+  </concept>
+
   <concept id="scalability_hotspots" rev="2.5.0 IMPALA-2696">
     <title>Avoiding CPU Hotspots for HDFS Cached Data</title>
     <conbody>


Mime
View raw message