Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4E14310D84 for ; Fri, 14 Feb 2014 16:59:29 +0000 (UTC) Received: (qmail 10944 invoked by uid 500); 14 Feb 2014 16:59:28 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 10916 invoked by uid 500); 14 Feb 2014 16:59:28 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 10904 invoked by uid 99); 14 Feb 2014 16:59:28 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Feb 2014 16:59:28 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id BFB1B1C0429; Fri, 14 Feb 2014 16:59:26 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============7141268712543618263==" MIME-Version: 1.0 Subject: Re: Review Request 18123: Custom log4j configs should be stored as raw text From: "Dmytro Sen" To: "Sumit Mohanty" , "Dmitro Lisnichenko" Cc: "Dmytro Sen" , "Ambari" Date: Fri, 14 Feb 2014 16:59:26 -0000 Message-ID: <20140214165926.29299.38560@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Dmytro Sen" X-ReviewGroup: Ambari X-ReviewRequest-URL: https://reviews.apache.org/r/18123/ X-Sender: "Dmytro Sen" References: <20140214161740.29299.39312@reviews.apache.org> In-Reply-To: <20140214161740.29299.39312@reviews.apache.org> Reply-To: "Dmytro Sen" X-ReviewRequest-Repository: ambari --===============7141268712543618263== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18123/ ----------------------------------------------------------- (Updated Feb. 14, 2014, 4:59 p.m.) Review request for Ambari, Dmitro Lisnichenko and Sumit Mohanty. Changes ------- Added unit tests for oozie, merged patch with current trunk Bugs: AMBARI-4678 https://issues.apache.org/jira/browse/AMBARI-4678 Repository: ambari Description ------- In 1.5.0, we are introducing customization of log4j.properties for various services ("hdfs-log4j" configuration, etc). Instead of storing the log4j configuration as a bunch of key-value pairs, we should store the entire log4j.properties file content as a single attribute (lets call it "content") whose value is the entire file content of log4j.properties in raw text. This would be much easier for the end user to view and modify. Also, this allows us to retain comments and logical breaks in the file. Actually we could reuse current design. We just have to store the log4g propertieas not as key-value pair, but as a raw text in a single "content" attribute. Steps to be completed to put log4. 1. UI downloads from the server log4j.properties as a raw text 2. User is able to customize this log4j configuration in textarea 3. UI escapes quotes(\"), new lines(\n), other special characters and puts the config like: {noformat} curl -H 'X-Requested-By: X-Requested-By' -u admin:admin -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"oozie-log4j","tag":"version1","properties":{"content":"line1\nline2\n"}}}}' http://localhost:8080/api/v1/clusters/c1 {noformat} 4. Ambari-agent stores the "content" attribute value to the spesified log4j.properties file using resource File like : {code} File(format("{params.conf_dir}/oozie-log4j.properties"), mode=0644, group=params.user_group, owner=params.oozie_user, content=params.configuration['oozie-log4j']['content'] ) {code} I've tested this design. It allows to store and deliver to the agent log4j.properties as a raw text with any comments or escaped characters. Diffs (updated) ----- ambari-server/src/main/resources/stacks/HDP/1.3.2/hooks/before-START/scripts/params.py 0b62c87 ambari-server/src/main/resources/stacks/HDP/1.3.2/hooks/before-START/scripts/shared_initialization.py d6521ee ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HBASE/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HBASE/package/scripts/hbase.py c140a6d ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HBASE/package/scripts/params.py 9722034 ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HIVE/configuration/hive-exec-log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HIVE/configuration/hive-log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HIVE/package/scripts/hive.py 940f8c3 ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HIVE/package/scripts/params.py b999aec ambari-server/src/main/resources/stacks/HDP/1.3.2/services/MAPREDUCE/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/OOZIE/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/OOZIE/package/scripts/oozie.py 151a1f8 ambari-server/src/main/resources/stacks/HDP/1.3.2/services/OOZIE/package/scripts/params.py 517ee73 ambari-server/src/main/resources/stacks/HDP/1.3.2/services/PIG/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/PIG/package/scripts/params.py 3cd60d2 ambari-server/src/main/resources/stacks/HDP/1.3.2/services/PIG/package/scripts/pig.py 0450d5e ambari-server/src/main/resources/stacks/HDP/1.3.2/services/ZOOKEEPER/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/1.3.2/services/ZOOKEEPER/configuration/zookeeper-log4j.xml 8fb48b9 ambari-server/src/main/resources/stacks/HDP/1.3.2/services/ZOOKEEPER/package/scripts/params.py 60fa5da ambari-server/src/main/resources/stacks/HDP/1.3.2/services/ZOOKEEPER/package/scripts/zookeeper.py 7b74b4c ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 3239c7a ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py 2f2ca8b ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase.py d899d17 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/package/scripts/params.py a9086e7 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/configuration/hive-exec-log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/configuration/hive-log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive.py bb1c065 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py ea51420 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/package/scripts/oozie.py 2efe535 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/package/scripts/params.py dc52b39 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/PIG/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/PIG/package/scripts/params.py 3cd60d2 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/PIG/package/scripts/pig.py d8aadf7 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/configuration/log4j.properties PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/configuration/zookeeper-log4j.xml 8fb48b9 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/package/scripts/params.py 60fa5da ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/package/scripts/zookeeper.py ac69829 ambari-server/src/test/python/stacks/1.3.2/HBASE/test_hbase_client.py 5ef19c3 ambari-server/src/test/python/stacks/1.3.2/HBASE/test_hbase_master.py 1d1baca ambari-server/src/test/python/stacks/1.3.2/HBASE/test_hbase_regionserver.py a031b04 ambari-server/src/test/python/stacks/1.3.2/HIVE/test_hive_client.py d1610d6 ambari-server/src/test/python/stacks/1.3.2/HIVE/test_hive_metastore.py c970cc8 ambari-server/src/test/python/stacks/1.3.2/HIVE/test_hive_server.py 45b7a0a ambari-server/src/test/python/stacks/1.3.2/OOZIE/test_oozie_client.py e60b1ad ambari-server/src/test/python/stacks/1.3.2/OOZIE/test_oozie_server.py 7093ac7 ambari-server/src/test/python/stacks/1.3.2/PIG/test_pig_client.py 1856be6 ambari-server/src/test/python/stacks/1.3.2/ZOOKEEPER/test_zookeeper_client.py f7de9f9 ambari-server/src/test/python/stacks/1.3.2/ZOOKEEPER/test_zookeeper_server.py 81df5e7 ambari-server/src/test/python/stacks/1.3.2/configs/default.json 653be3c ambari-server/src/test/python/stacks/1.3.2/configs/secured.json e711ff2 ambari-server/src/test/python/stacks/1.3.2/hooks/before-START/test_before_start.py d3b7788 ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_client.py bd3228f ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py ef58f17 ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_regionserver.py 3efe21e ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_client.py ec3c895 ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py c08a260 ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 071d108 ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py 358cab9 ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 8e35e23 ambari-server/src/test/python/stacks/2.0.6/PIG/test_pig_client.py 0143e27 ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_client.py db1b883 ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_server.py f5ac3eb ambari-server/src/test/python/stacks/2.0.6/configs/default.json d3d5ba3 ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 61a4830 ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py 6045f39 Diff: https://reviews.apache.org/r/18123/diff/ Testing (updated) ------- Fixed unit tests [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ....................................... SUCCESS [0.069s] [INFO] Apache Ambari Project POM ......................... SUCCESS [0.026s] [INFO] Ambari Web ........................................ SUCCESS [11.989s] [INFO] Ambari Views ...................................... SUCCESS [1.771s] [INFO] Ambari Server ..................................... SUCCESS [11:08.445s] [INFO] Ambari Agent ...................................... SUCCESS [12.989s] [INFO] Ambari Client ..................................... SUCCESS [0.511s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 11:38.352s [INFO] Finished at: Fri Feb 14 19:52:25 FET 2014 [INFO] Final Memory: 159M/932M Thanks, Dmytro Sen --===============7141268712543618263==--