Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 543FF200B34 for ; Sat, 18 Jun 2016 00:18:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 52B9E160A62; Fri, 17 Jun 2016 22:18:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A6385160A61 for ; Sat, 18 Jun 2016 00:18:56 +0200 (CEST) Received: (qmail 38200 invoked by uid 500); 17 Jun 2016 22:18:55 -0000 Mailing-List: contact commits-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: ambari-dev@ambari.apache.org Delivered-To: mailing list commits@ambari.apache.org Received: (qmail 38191 invoked by uid 99); 17 Jun 2016 22:18:55 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2016 22:18:55 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 52E8AE049D; Fri, 17 Jun 2016 22:18:55 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: alejandro@apache.org To: commits@ambari.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: ambari git commit: AMBARI-17274. As part of atlas startup, '/usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh' script is executed by ambari, which fails with 'java.io.FileNotFoundException' (Tom Beerbower via alejandro) Date: Fri, 17 Jun 2016 22:18:55 +0000 (UTC) archived-at: Fri, 17 Jun 2016 22:18:58 -0000 Repository: ambari Updated Branches: refs/heads/trunk 54d76f5b8 -> 92237462d AMBARI-17274. As part of atlas startup, '/usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh' script is executed by ambari, which fails with 'java.io.FileNotFoundException' (Tom Beerbower via alejandro) Project: http://git-wip-us.apache.org/repos/asf/ambari/repo Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/92237462 Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/92237462 Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/92237462 Branch: refs/heads/trunk Commit: 92237462de2e5f4177113a07129b10834a9c5ca0 Parents: 54d76f5 Author: Alejandro Fernandez Authored: Fri Jun 17 15:16:39 2016 -0700 Committer: Alejandro Fernandez Committed: Fri Jun 17 15:20:32 2016 -0700 ---------------------------------------------------------------------- .../configuration/atlas-solrconfig.xml | 641 +++++++++++++++++++ .../ATLAS/0.1.0.2.3/metainfo.xml | 11 + .../ATLAS/0.1.0.2.3/package/scripts/metadata.py | 17 +- .../ATLAS/0.1.0.2.3/package/scripts/params.py | 2 + .../stacks/2.3/ATLAS/test_metadata_server.py | 17 +- .../test/python/stacks/2.3/configs/default.json | 4 +- .../test/python/stacks/2.3/configs/secure.json | 4 +- .../stacks/2.5/ATLAS/test_atlas_server.py | 17 +- .../test/python/stacks/2.5/configs/default.json | 4 +- 9 files changed, 708 insertions(+), 9 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml new file mode 100644 index 0000000..35e9f58 --- /dev/null +++ b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml @@ -0,0 +1,641 @@ + + + + + + content + atlas-solrconfig template + Atlas Solr configuration + + + + + + + + + 5.5.1 + + + ${solr.data.dir:} + + + + + + + + + + + + true + managed-schema + + + + + + + ${solr.lock.type:native} + + + true + + + + + + + + + + + + + + + + ${solr.ulog.dir:} + ${solr.ulog.numVersionBuckets:65536} + + + + + ${solr.autoCommit.maxTime:15000} + false + + + + + ${solr.autoSoftCommit.maxTime:-1} + + + + + + + + 1024 + + + + + + + + + + + + + + + + + + true + + + 20 + + + 200 + + + false + + + 2 + + + + + + + + + + + + + + + + + + + + explicit + 10 + + + + + + + + explicit + json + true + text + + + + + + + {!xport} + xsort + false + + + + query + + + + + + + text + + + + + + + + + + + + + + explicit + true + + + + + + + + + + + + + + true + false + + + terms + + + + + + *:* + + + + ]]> + + content + false + + + + http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml index 01276a6..b4df814 100644 --- a/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml +++ b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml @@ -59,6 +59,11 @@ atlas-log4j.xml atlas-log4j + + env + atlas-solrconfig.xml + atlas-solrconfig + @@ -88,6 +93,11 @@ atlas-log4j.xml atlas-log4j + + env + atlas-solrconfig.xml + atlas-solrconfig + @@ -117,6 +127,7 @@ atlas-env atlas-log4j core-site + atlas-solrconfig http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py index 1abf2c1..5ace399 100644 --- a/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py +++ b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py @@ -46,6 +46,14 @@ def metadata(type='server'): create_parents = True ) + Directory(format('{conf_dir}/solr'), + mode=0755, + cd_access='a', + owner=params.metadata_user, + group=params.user_group, + create_parents = True + ) + Directory(params.log_dir, mode=0755, cd_access='a', @@ -95,6 +103,13 @@ def metadata(type='server'): content=InlineTemplate(params.metadata_log4j_content) ) + File(format("{conf_dir}/solr/solrconfig.xml"), + mode=0644, + owner=params.metadata_user, + group=params.user_group, + content=InlineTemplate(params.metadata_solrconfig_content) + ) + if type == 'server' and params.search_backend_solr and params.has_logsearch_solr: solr_cloud_util.setup_solr_client(params.config) @@ -117,7 +132,7 @@ def upload_conf_set(config_set, random_num): solr_cloud_util.upload_configuration_to_zk( zookeeper_quorum=params.zookeeper_quorum, solr_znode=params.logsearch_solr_znode, - config_set_dir=format("{logsearch_solr_dir}/server/solr/configsets/{config_set}/conf"), + config_set_dir=format("{conf_dir}/solr"), config_set=config_set, tmp_config_set_dir=tmp_config_set_folder, java64_home=params.java64_home, http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py index 614fa6b..8513c96 100644 --- a/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py +++ b/ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py @@ -111,6 +111,8 @@ expanded_war_dir = os.environ['METADATA_EXPANDED_WEBAPP_DIR'] if 'METADATA_EXPAN metadata_log4j_content = config['configurations']['atlas-log4j']['content'] +metadata_solrconfig_content = config['configurations']['atlas-solrconfig']['content'] + atlas_log_level = config['configurations']['atlas-log4j']['atlas_log_level'] audit_log_level = config['configurations']['atlas-log4j']['audit_log_level'] http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py b/ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py index de355ea..f627066 100644 --- a/ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py +++ b/ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py @@ -45,6 +45,13 @@ class TestMetadataServer(RMFTestCase): cd_access='a', mode=0755 ) + self.assertResourceCalled('Directory', '/etc/atlas/conf/solr', + owner='atlas', + group='hadoop', + create_parents = True, + cd_access='a', + mode=0755 + ) self.assertResourceCalled('Directory', '/var/log/atlas', owner='atlas', group='hadoop', @@ -96,6 +103,14 @@ class TestMetadataServer(RMFTestCase): group='hadoop', mode=0644, ) + self.assertResourceCalled('File', '/etc/atlas/conf/solr/solrconfig.xml', + content=InlineTemplate( + self.getConfig()['configurations'][ + 'atlas-solrconfig']['content']), + owner='atlas', + group='hadoop', + mode=0644, + ) self.assertResourceCalled('Directory', '/var/log/ambari-logsearch-solr-client', owner='solr', group='hadoop', @@ -131,7 +146,7 @@ class TestMetadataServer(RMFTestCase): content = '' ) self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --download-config -d /tmp/solr_config_basic_configs_0.[0-9]* -cs basic_configs -rt 5 -i 10') - self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --upload-config -d /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs basic_configs -rt 5 -i 10') + self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --upload-config -d /etc/atlas/conf/solr -cs basic_configs -rt 5 -i 10') self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --create-collection -c vertex_index -cs basic_configs -s 1 -r 1 -m 1 -rt 5 -i 10') self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --create-collection -c edge_index -cs basic_configs -s 1 -r 1 -m 1 -rt 5 -i 10') http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/test/python/stacks/2.3/configs/default.json ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/stacks/2.3/configs/default.json b/ambari-server/src/test/python/stacks/2.3/configs/default.json index 07298f1..d8fb398 100644 --- a/ambari-server/src/test/python/stacks/2.3/configs/default.json +++ b/ambari-server/src/test/python/stacks/2.3/configs/default.json @@ -226,8 +226,8 @@ "atlas_log_level": "debug", "audit_log_level": "OFF" }, - "atlas-hbase-site": { - "hbase.master.info.port": "61510" + "atlas-solrconfig": { + "content": "contentCustom solrconfig properties" }, "ranger-hbase-plugin-properties": { "ranger-hbase-plugin-enabled":"yes" http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/test/python/stacks/2.3/configs/secure.json ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/stacks/2.3/configs/secure.json b/ambari-server/src/test/python/stacks/2.3/configs/secure.json index 1ca7a2e..338f048 100644 --- a/ambari-server/src/test/python/stacks/2.3/configs/secure.json +++ b/ambari-server/src/test/python/stacks/2.3/configs/secure.json @@ -218,8 +218,8 @@ "atlas_log_level": "debug", "audit_log_level": "OFF" }, - "atlas-hbase-site": { - "hbase.master.info.port": "61510" + "atlas-solrconfig": { + "content": "contentCustom solrconfig properties" }, "ranger-hbase-plugin-properties": { "ranger-hbase-plugin-enabled":"yes" http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py b/ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py index 8729195..6d226a6 100644 --- a/ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py +++ b/ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py @@ -42,6 +42,13 @@ class TestAtlasServer(RMFTestCase): cd_access='a', mode=0755 ) + self.assertResourceCalled('Directory', '/etc/atlas/conf/solr', + owner='atlas', + group='hadoop', + create_parents = True, + cd_access='a', + mode=0755 + ) self.assertResourceCalled('Directory', '/var/log/atlas', owner='atlas', group='hadoop', @@ -94,6 +101,14 @@ class TestAtlasServer(RMFTestCase): group='hadoop', mode=0644, ) + self.assertResourceCalled('File', '/etc/atlas/conf/solr/solrconfig.xml', + content=InlineTemplate( + self.getConfig()['configurations'][ + 'atlas-solrconfig']['content']), + owner='atlas', + group='hadoop', + mode=0644, + ) self.assertResourceCalled('Directory', '/var/log/ambari-logsearch-solr-client', owner='solr', group='hadoop', @@ -130,7 +145,7 @@ class TestAtlasServer(RMFTestCase): ) self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --download-config -d /tmp/solr_config_basic_configs_0.[0-9]* -cs basic_configs -rt 5 -i 10') - self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --upload-config -d /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs basic_configs -rt 5 -i 10') + self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --upload-config -d /etc/atlas/conf/solr -cs basic_configs -rt 5 -i 10') self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --create-collection -c vertex_index -cs basic_configs -s 1 -r 1 -m 1 -rt 5 -i 10') self.assertResourceCalledRegexp('^Execute$', '^export JAVA_HOME=/usr/jdk64/jdk1.7.0_45 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z c6401.ambari.apache.org/logsearch --create-collection -c edge_index -cs basic_configs -s 1 -r 1 -m 1 -rt 5 -i 10') http://git-wip-us.apache.org/repos/asf/ambari/blob/92237462/ambari-server/src/test/python/stacks/2.5/configs/default.json ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/stacks/2.5/configs/default.json b/ambari-server/src/test/python/stacks/2.5/configs/default.json index c839405..de05d76 100644 --- a/ambari-server/src/test/python/stacks/2.5/configs/default.json +++ b/ambari-server/src/test/python/stacks/2.5/configs/default.json @@ -225,8 +225,8 @@ "atlas_log_level": "debug", "audit_log_level": "OFF" }, - "atlas-hbase-site": { - "hbase.master.info.port": "61510" + "atlas-solrconfig": { + "content": "contentCustom solrconfig properties" }, "ranger-hbase-plugin-properties": { "ranger-hbase-plugin-enabled":"yes"