Return-Path: X-Original-To: apmail-ambari-commits-archive@www.apache.org Delivered-To: apmail-ambari-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B4E01181EA for ; Wed, 27 Jan 2016 12:08:48 +0000 (UTC) Received: (qmail 76027 invoked by uid 500); 27 Jan 2016 12:08:42 -0000 Delivered-To: apmail-ambari-commits-archive@ambari.apache.org Received: (qmail 75991 invoked by uid 500); 27 Jan 2016 12:08:42 -0000 Mailing-List: contact commits-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: ambari-dev@ambari.apache.org Delivered-To: mailing list commits@ambari.apache.org Received: (qmail 75982 invoked by uid 99); 27 Jan 2016 12:08:42 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jan 2016 12:08:42 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 1FFBFE00D6; Wed, 27 Jan 2016 12:08:42 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: aonishuk@apache.org To: commits@ambari.apache.org Date: Wed, 27 Jan 2016 12:08:42 -0000 Message-Id: <14a94462183a4c78bbdba7987598192e@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [1/2] ambari git commit: AMBARI-14808. Add Oozie limits.conf for Oozie User as done for HDFS/YARN (aonishuk) Repository: ambari Updated Branches: refs/heads/branch-2.2 39e0eb20a -> 7d188b63a refs/heads/trunk 34bbd8b45 -> a88678fc7 AMBARI-14808. Add Oozie limits.conf for Oozie User as done for HDFS/YARN (aonishuk) Project: http://git-wip-us.apache.org/repos/asf/ambari/repo Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/a88678fc Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/a88678fc Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/a88678fc Branch: refs/heads/trunk Commit: a88678fc7d3aeba814236d53ad9c98803f3cf4b8 Parents: 34bbd8b Author: Andrew Onishuk Authored: Wed Jan 27 14:08:30 2016 +0200 Committer: Andrew Onishuk Committed: Wed Jan 27 14:08:30 2016 +0200 ---------------------------------------------------------------------- .../OOZIE/4.0.0.2.0/configuration/oozie-env.xml | 10 +++++ .../OOZIE/4.0.0.2.0/package/scripts/oozie.py | 14 +++++++ .../4.0.0.2.0/package/scripts/params_linux.py | 5 +++ .../4.0.0.2.0/package/templates/oozie.conf.j2 | 35 ++++++++++++++++ .../stacks/2.0.6/OOZIE/test_oozie_client.py | 33 +++++++++++++++ .../stacks/2.0.6/OOZIE/test_oozie_server.py | 44 ++++++++++++++++++++ 6 files changed, 141 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/ambari/blob/a88678fc/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml index 3db5463..fae449e 100644 --- a/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml +++ b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml @@ -98,6 +98,16 @@ 256 Oozie permanent generation size. + + oozie_user_nofile_limit + 32000 + Max open files limit setting for OOZIE user. + + + oozie_user_nproc_limit + 16000 + Max number of processes limit setting for OOZIE user. + http://git-wip-us.apache.org/repos/asf/ambari/blob/a88678fc/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py index db1b224..5fc2cfb 100644 --- a/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py +++ b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py @@ -114,6 +114,20 @@ def oozie(is_server=False): group=params.user_group, ) + # On some OS this folder could be not exists, so we will create it before pushing there files + Directory(params.limits_conf_dir, + recursive=True, + owner='root', + group='root' + ) + + File(os.path.join(params.limits_conf_dir, 'oozie.conf'), + owner='root', + group='root', + mode=0644, + content=Template("oozie.conf.j2") + ) + if (params.log4j_props != None): File(format("{params.conf_dir}/oozie-log4j.properties"), mode=0644, http://git-wip-us.apache.org/repos/asf/ambari/blob/a88678fc/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py index fd9a462..a8a93dd 100644 --- a/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py +++ b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py @@ -124,6 +124,11 @@ security_enabled = config['configurations']['cluster-env']['security_enabled'] oozie_heapsize = config['configurations']['oozie-env']['oozie_heapsize'] oozie_permsize = config['configurations']['oozie-env']['oozie_permsize'] +limits_conf_dir = "/etc/security/limits.d" + +oozie_user_nofile_limit = config['configurations']['oozie-env']['oozie_user_nofile_limit'] +oozie_user_nproc_limit = config['configurations']['oozie-env']['oozie_user_nproc_limit'] + kinit_path_local = get_kinit_path(default('/configurations/kerberos-env/executable_search_paths', None)) oozie_service_keytab = config['configurations']['oozie-site']['oozie.service.HadoopAccessorService.keytab.file'] oozie_principal = config['configurations']['oozie-site']['oozie.service.HadoopAccessorService.kerberos.principal'] http://git-wip-us.apache.org/repos/asf/ambari/blob/a88678fc/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2 ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2 b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2 new file mode 100644 index 0000000..1f99e49 --- /dev/null +++ b/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2 @@ -0,0 +1,35 @@ +{# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +#} + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{oozie_user}} - nofile {{oozie_user_nofile_limit}} +{{oozie_user}} - nproc {{oozie_user_nproc_limit}} http://git-wip-us.apache.org/repos/asf/ambari/blob/a88678fc/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py b/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py index ce63b80..4c1a9bc 100644 --- a/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py +++ b/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py @@ -52,6 +52,17 @@ class TestOozieClient(RMFTestCase): content = InlineTemplate(self.getConfig()['configurations']['oozie-env']['content']), group = 'hadoop', ) + self.assertResourceCalled('Directory', '/etc/security/limits.d', + owner = 'root', + group = 'root', + recursive=True, + ) + self.assertResourceCalled('File', '/etc/security/limits.d/oozie.conf', + owner = 'root', + group = 'root', + mode=0644, + content=Template("oozie.conf.j2"), + ) self.assertResourceCalled('File', '/etc/oozie/conf/oozie-log4j.properties', owner = 'oozie', group = 'hadoop', @@ -107,6 +118,17 @@ class TestOozieClient(RMFTestCase): content = InlineTemplate(self.getConfig()['configurations']['oozie-env']['content']), group = 'hadoop', ) + self.assertResourceCalled('Directory', '/etc/security/limits.d', + owner = 'root', + group = 'root', + recursive=True, + ) + self.assertResourceCalled('File', '/etc/security/limits.d/oozie.conf', + owner = 'root', + group = 'root', + mode=0644, + content=Template("oozie.conf.j2"), + ) self.assertResourceCalled('File', '/etc/oozie/conf/oozie-log4j.properties', owner = 'oozie', group = 'hadoop', @@ -168,6 +190,17 @@ class TestOozieClient(RMFTestCase): content = InlineTemplate(self.getConfig()['configurations']['oozie-env']['content']), group = 'hadoop', ) + self.assertResourceCalled('Directory', '/etc/security/limits.d', + owner = 'root', + group = 'root', + recursive=True, + ) + self.assertResourceCalled('File', '/etc/security/limits.d/oozie.conf', + owner = 'root', + group = 'root', + mode=0644, + content=Template("oozie.conf.j2"), + ) self.assertResourceCalled('File', '/usr/hdp/current/oozie-client/conf/oozie-log4j.properties', owner = 'oozie', group = 'hadoop', http://git-wip-us.apache.org/repos/asf/ambari/blob/a88678fc/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py b/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py index 33facf7..d167ef3 100644 --- a/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py +++ b/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py @@ -105,6 +105,17 @@ class TestOozieServer(RMFTestCase): owner = 'oozie', group = 'hadoop', ) + self.assertResourceCalled('Directory', '/etc/security/limits.d', + owner = 'root', + group = 'root', + recursive=True, + ) + self.assertResourceCalled('File', '/etc/security/limits.d/oozie.conf', + owner = 'root', + group = 'root', + mode=0644, + content=Template("oozie.conf.j2"), + ) self.assertResourceCalled('File', '/etc/oozie/conf/oozie-log4j.properties', content = 'log4jproperties\nline2', owner = 'oozie', @@ -313,6 +324,17 @@ class TestOozieServer(RMFTestCase): owner = 'oozie', group = 'hadoop', ) + self.assertResourceCalled('Directory', '/etc/security/limits.d', + owner = 'root', + group = 'root', + recursive=True, + ) + self.assertResourceCalled('File', '/etc/security/limits.d/oozie.conf', + owner = 'root', + group = 'root', + mode=0644, + content=Template("oozie.conf.j2"), + ) self.assertResourceCalled('File', '/etc/oozie/conf/oozie-log4j.properties', content = 'log4jproperties\nline2', owner = 'oozie', @@ -673,6 +695,17 @@ class TestOozieServer(RMFTestCase): content = InlineTemplate(self.getConfig()['configurations']['oozie-env']['content']), group = 'hadoop', ) + self.assertResourceCalled('Directory', '/etc/security/limits.d', + owner = 'root', + group = 'root', + recursive=True, + ) + self.assertResourceCalled('File', '/etc/security/limits.d/oozie.conf', + owner = 'root', + group = 'root', + mode=0644, + content=Template("oozie.conf.j2"), + ) self.assertResourceCalled('File', '/etc/oozie/conf/oozie-log4j.properties', owner = 'oozie', group = 'hadoop', @@ -852,6 +885,17 @@ class TestOozieServer(RMFTestCase): content = InlineTemplate(self.getConfig()['configurations']['oozie-env']['content']), group = 'hadoop', ) + self.assertResourceCalled('Directory', '/etc/security/limits.d', + owner = 'root', + group = 'root', + recursive=True, + ) + self.assertResourceCalled('File', '/etc/security/limits.d/oozie.conf', + owner = 'root', + group = 'root', + mode=0644, + content=Template("oozie.conf.j2"), + ) self.assertResourceCalled('File', '/etc/oozie/conf/oozie-log4j.properties', owner = 'oozie', group = 'hadoop',