From commits-return-8678-archive-asf-public=cust-asf.ponee.io@kudu.apache.org Wed Mar 25 21:41:47 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EB27618063D for ; Wed, 25 Mar 2020 22:41:46 +0100 (CET) Received: (qmail 13866 invoked by uid 500); 25 Mar 2020 21:41:46 -0000 Mailing-List: contact commits-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kudu.apache.org Delivered-To: mailing list commits@kudu.apache.org Received: (qmail 13857 invoked by uid 99); 25 Mar 2020 21:41:46 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Mar 2020 21:41:46 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id 22994819F7; Wed, 25 Mar 2020 21:41:46 +0000 (UTC) Date: Wed, 25 Mar 2020 21:41:46 +0000 To: "commits@kudu.apache.org" Subject: [kudu] branch master updated: KUDU-3079 Add MiniRanger MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <158517250605.18371.17057400771702552712@gitbox.apache.org> From: awong@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: kudu X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Oldrev: c0cb960bd39fcbf573247596e81acf5068b924db X-Git-Newrev: b417479266ef0f09570d4536738ca18cf48d7ff2 X-Git-Rev: b417479266ef0f09570d4536738ca18cf48d7ff2 X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/master by this push: new b417479 KUDU-3079 Add MiniRanger b417479 is described below commit b417479266ef0f09570d4536738ca18cf48d7ff2 Author: Attila Bukor AuthorDate: Wed Mar 25 21:47:36 2020 +0100 KUDU-3079 Add MiniRanger Apache Ranger has some setup scripts[1][2] that should be used to install it but they touch a lot of things on the file system that unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've opted for a more manual approach similar to what Impala have chosen[3]. As Ranger 2.0.0 was released back in August 2019 which didn't contain the Kudu service definition and it wasn't a clean backport I opted to use the tip of the master (at the time of writing). Unfortunately Ranger's scripts were written for Python 2 and there were several compatibility issues with Python 3 so I had to patch it. Most of the changes were generated by Python's "2to3" script but it required some manual changes as well to run on Python 3 (subprocesses return bytes and they were compared with strings which wasn't a problem in Python 2 but fails in Python 3 so the subprocess result had to be decoded). I had to apply another patch to fix an issue running db_setup.py on Mac as it runs ranger-admin-services.sh which sets the script path by running `readlink -f $0` which doesn't work on Mac. Instead of rewriting the script in a way that would work properly on Mac and Linux I opted to use an environment variable instead, so I decided to use a separate patch file for this as this one will need to be applied even after Ranger decides to support Python 3[4]. The commit also adds support for HTTP Basic auth in EasyCurl along with specifying headers in POST requests which is used in MiniRanger to talk to the Ranger admin REST API when creating the Kudu service after startup and adding policies. Also adds a CreateSymLink() method to Env. Starting MiniRanger (fresh install) takes ~20s on my machine (2019 MacBook Pro) and ~30-40s on dist-test server. Starting Ranger subprocess is not handled in this commit, it is going to be done in a follow-up patch. Thanks to Andrew Wong for his help with the Ranger setup[5]. [1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh [2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh [3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22 [4] https://issues.apache.org/jira/browse/RANGER-2759 [5] https://gerrit.cloudera.org/c/15385/5 Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6 Reviewed-on: http://gerrit.cloudera.org:8080/15483 Reviewed-by: Adar Dembo Tested-by: Attila Bukor Reviewed-by: Andrew Wong --- build-support/dist_test.py | 3 +- build-support/run_dist_test.py | 17 +- src/kudu/mini-cluster/CMakeLists.txt | 1 + src/kudu/mini-cluster/external_mini_cluster.cc | 7 + src/kudu/mini-cluster/external_mini_cluster.h | 14 + src/kudu/postgres/mini_postgres.cc | 9 +- src/kudu/ranger/CMakeLists.txt | 33 +- src/kudu/ranger/mini_ranger-test.cc | 84 ++++ src/kudu/ranger/mini_ranger.cc | 306 ++++++++++++ src/kudu/ranger/mini_ranger.h | 154 ++++++ src/kudu/ranger/mini_ranger_configs.h | 349 +++++++++++++ src/kudu/security/test/test_pass.h | 5 +- src/kudu/server/webserver-test.cc | 9 +- src/kudu/tools/tool.proto | 3 + src/kudu/tools/tool_action.h | 1 + src/kudu/tools/tool_action_test.cc | 4 +- src/kudu/util/curl_util.cc | 37 +- src/kudu/util/curl_util.h | 28 +- src/kudu/util/env-test.cc | 15 + src/kudu/util/env.h | 3 + src/kudu/util/env_posix.cc | 11 + thirdparty/LICENSE.txt | 5 + thirdparty/build-thirdparty.sh | 14 + thirdparty/download-thirdparty.sh | 8 + thirdparty/patches/ranger-fixscripts.patch | 14 + thirdparty/patches/ranger-python3.patch | 667 +++++++++++++++++++++++++ thirdparty/vars.sh | 9 + 27 files changed, 1779 insertions(+), 31 deletions(-) diff --git a/build-support/dist_test.py b/build-support/dist_test.py index 5056e83..aa20d70 100755 --- a/build-support/dist_test.py +++ b/build-support/dist_test.py @@ -113,11 +113,12 @@ DEPS_FOR_ALL = \ # Add the Kudu echo subprocess. "build/latest/bin/kudu-subprocess.jar", - # Add Postgres. These are symlinks to directories in thirdparty. + # Add Postgres and Ranger. These are symlinks to directories in thirdparty. "build/latest/bin/postgres", "build/latest/bin/postgres-lib", "build/latest/bin/postgres-share", "build/latest/bin/postgresql.jar", + "build/latest/bin/ranger-home", # Add the Kudu HMS plugin. "build/latest/bin/hms-plugin.jar", diff --git a/build-support/run_dist_test.py b/build-support/run_dist_test.py index 09404b9..cfb2c75 100755 --- a/build-support/run_dist_test.py +++ b/build-support/run_dist_test.py @@ -151,10 +151,12 @@ def main(): env['HIVE_HOME'] = glob.glob(os.path.join(ROOT, "thirdparty/src/hive-*"))[0] env['HADOOP_HOME'] = glob.glob(os.path.join(ROOT, "thirdparty/src/hadoop-*"))[0] env['SENTRY_HOME'] = glob.glob(os.path.join(ROOT, "thirdparty/src/sentry-*"))[0] + env['RANGER_HOME'] = glob.glob(os.path.join(ROOT, "thirdparty/src/ranger-*"))[0] env['JAVA_HOME'] = glob.glob("/usr/lib/jvm/java-1.8.0-*")[0] - # Restore the symlinks to the chrony binaries and Postgres directories; tests - # expect to find them in same directory as the test binaries themselves. + # Restore the symlinks to the chrony binaries and Postgres and Ranger + # directories; tests expect to find them in same directory as the test + # binaries themselves. for bin_path in glob.glob(os.path.join(ROOT, "build/*/bin")): os.symlink(os.path.join(ROOT, "thirdparty/installed/common/bin/chronyc"), os.path.join(bin_path, "chronyc")) @@ -166,8 +168,17 @@ def main(): os.path.join(bin_path, "postgres-lib")) os.symlink(os.path.join(ROOT, "thirdparty/installed/common/share/postgresql"), os.path.join(bin_path, "postgres-share")) - os.symlink(os.path.join(ROOT, "thirdparty/installed/common/opt/jdbc/postgresql.jar"), + os.symlink(glob.glob(os.path.join(ROOT, "thirdparty/src/postgresql-*/postgresql-*.jar"))[0], os.path.join(bin_path, "postgresql.jar")) + os.symlink(glob.glob(os.path.join(ROOT, "thirdparty/src/ranger-*"))[0], + os.path.join(bin_path, "ranger-home")) + os.symlink(os.path.join(ROOT, "thirdparty/installed/common/opt/hadoop"), + os.path.join(bin_path, "hadoop-home")) + # When building Ranger, we symlink conf.dist to conf. Overwrite the link we + # copied over with a link that's suitable for the remote machine. + os.unlink(os.path.join(bin_path, "ranger-home/ews/webapp/WEB-INF/classes/conf")) + os.symlink(os.path.join(bin_path, "ranger-home/ews/webapp/WEB-INF/classes/conf.dist"), + os.path.join(bin_path, "ranger-home/ews/webapp/WEB-INF/classes/conf")) env['LD_LIBRARY_PATH'] = ":".join( [os.path.join(ROOT, "build/dist-test-system-libs/")] + diff --git a/src/kudu/mini-cluster/CMakeLists.txt b/src/kudu/mini-cluster/CMakeLists.txt index 63f6eea..93dbe74 100644 --- a/src/kudu/mini-cluster/CMakeLists.txt +++ b/src/kudu/mini-cluster/CMakeLists.txt @@ -36,6 +36,7 @@ set(MINI_CLUSTER_LIBS master_proto mini_hms mini_kdc + mini_ranger mini_sentry server_base_proto tablet_proto diff --git a/src/kudu/mini-cluster/external_mini_cluster.cc b/src/kudu/mini-cluster/external_mini_cluster.cc index 23c40c5..32fce60 100644 --- a/src/kudu/mini-cluster/external_mini_cluster.cc +++ b/src/kudu/mini-cluster/external_mini_cluster.cc @@ -47,6 +47,7 @@ #include "kudu/hms/mini_hms.h" #include "kudu/master/master.pb.h" #include "kudu/master/master.proxy.h" +#include "kudu/ranger/mini_ranger.h" #include "kudu/rpc/messenger.h" #include "kudu/rpc/rpc_controller.h" #include "kudu/rpc/sasl_common.h" @@ -118,6 +119,7 @@ ExternalMiniClusterOptions::ExternalMiniClusterOptions() enable_kerberos(false), hms_mode(HmsMode::NONE), enable_sentry(false), + enable_ranger(false), logtostderr(true), start_process_timeout(MonoDelta::FromSeconds(70)), rpc_negotiation_timeout(MonoDelta::FromSeconds(3)) @@ -332,6 +334,11 @@ Status ExternalMiniCluster::Start() { } } + if (opts_.enable_ranger) { + ranger_.reset(new ranger::MiniRanger(cluster_root())); + RETURN_NOT_OK_PREPEND(ranger_->Start(), "Failed to start the Ranger service"); + } + // Start the HMS. if (opts_.hms_mode == HmsMode::DISABLE_HIVE_METASTORE || opts_.hms_mode == HmsMode::ENABLE_HIVE_METASTORE || diff --git a/src/kudu/mini-cluster/external_mini_cluster.h b/src/kudu/mini-cluster/external_mini_cluster.h index fdd8cc4..379a157 100644 --- a/src/kudu/mini-cluster/external_mini_cluster.h +++ b/src/kudu/mini-cluster/external_mini_cluster.h @@ -77,6 +77,10 @@ namespace sentry { class MiniSentry; } // namespace sentry +namespace ranger { +class MiniRanger; +} // namespace ranger + namespace server { class ServerStatusPB; } // namespace server @@ -196,6 +200,11 @@ struct ExternalMiniClusterOptions { // Default: false. bool enable_sentry; + // If true, set up a Ranger service as part of this ExternalMiniCluster. + // + // Default: false. + bool enable_ranger; + // If true, sends logging output to stderr instead of a log file. // // Default: true. @@ -351,6 +360,10 @@ class ExternalMiniCluster : public MiniCluster { return sentry_.get(); } + ranger::MiniRanger* ranger() const { + return ranger_.get(); + } + const std::string& cluster_root() const { return opts_.cluster_root; } @@ -484,6 +497,7 @@ class ExternalMiniCluster : public MiniCluster { std::unique_ptr kdc_; std::unique_ptr hms_; std::unique_ptr sentry_; + std::unique_ptr ranger_; std::shared_ptr messenger_; diff --git a/src/kudu/postgres/mini_postgres.cc b/src/kudu/postgres/mini_postgres.cc index 9dcd898..1471db2 100644 --- a/src/kudu/postgres/mini_postgres.cc +++ b/src/kudu/postgres/mini_postgres.cc @@ -50,6 +50,9 @@ MiniPostgres::~MiniPostgres() { } Status MiniPostgres::Start() { + if (process_) { + return Status::IllegalState("Postgres already running"); + } Env* env = Env::Default(); VLOG(1) << "Starting Postgres"; @@ -94,7 +97,11 @@ Status MiniPostgres::Start() { } Status MiniPostgres::Stop() { - return process_->KillAndWait(SIGTERM); + if (process_) { + RETURN_NOT_OK(process_->KillAndWait(SIGTERM)); + process_.reset(); + } + return Status::OK(); } Status MiniPostgres::AddUser(const string& user, bool super) { diff --git a/src/kudu/ranger/CMakeLists.txt b/src/kudu/ranger/CMakeLists.txt index 2ad305e..a52040b 100644 --- a/src/kudu/ranger/CMakeLists.txt +++ b/src/kudu/ranger/CMakeLists.txt @@ -32,9 +32,9 @@ target_link_libraries(ranger_proto protobuf ) -############################## +####################################### # kudu_ranger -############################## +####################################### set(RANGER_SRCS ranger_client.cc) @@ -48,11 +48,38 @@ add_library(kudu_ranger ${RANGER_SRCS}) target_link_libraries(kudu_ranger ${RANGER_DEPS}) ####################################### +# mini_ranger +####################################### + +execute_process(COMMAND ln -nsf + "${CMAKE_SOURCE_DIR}/thirdparty/installed/common/opt/ranger" + "${EXECUTABLE_OUTPUT_PATH}/ranger-home") +execute_process(COMMAND ln -nsf + "${CMAKE_SOURCE_DIR}/thirdparty/installed/common/opt/hadoop" + "${EXECUTABLE_OUTPUT_PATH}/hadoop-home") +execute_process(COMMAND ln -nsf + "${JAVA_HOME}" + "${EXECUTABLE_OUTPUT_PATH}/java-home") + +set(MINI_RANGER_SRCS + mini_ranger.cc) +set(MINI_RANGER_DEPS + kudu_curl_util + kudu_test_util + kudu_ranger + kudu_util + mini_postgres) +add_library(mini_ranger ${MINI_RANGER_SRCS}) +target_link_libraries(mini_ranger ${MINI_RANGER_DEPS}) + +####################################### # Unit tests ####################################### SET_KUDU_TEST_LINK_LIBS( kudu_ranger - mini_postgres) + mini_postgres + mini_ranger) +ADD_KUDU_TEST(mini_ranger-test) ADD_KUDU_TEST(ranger_client-test) diff --git a/src/kudu/ranger/mini_ranger-test.cc b/src/kudu/ranger/mini_ranger-test.cc new file mode 100644 index 0000000..42fc271 --- /dev/null +++ b/src/kudu/ranger/mini_ranger-test.cc @@ -0,0 +1,84 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "kudu/ranger/mini_ranger.h" + +#include +#include +#include + +#include + +#include "kudu/ranger/ranger.pb.h" +#include "kudu/util/status.h" +#include "kudu/util/test_macros.h" +#include "kudu/util/test_util.h" + +using std::string; + +namespace kudu { +namespace ranger { + +class MiniRangerTest : public KuduTest { + public: + void SetUp() override { + ASSERT_OK(ranger_.Start()); + } + + protected: + MiniRanger ranger_; +}; + +TEST_F(MiniRangerTest, TestGrantPrivilege) { + PolicyItem item; + item.first.emplace_back("testuser"); + item.second.emplace_back(ActionPB::ALTER); + + AuthorizationPolicy policy; + policy.databases.emplace_back("foo"); + policy.tables.emplace_back("bar"); + policy.items.emplace_back(std::move(item)); + policy.name = "test1"; + + ASSERT_OK(ranger_.AddPolicy(std::move(policy))); +} + +TEST_F(MiniRangerTest, TestGrantSamePrivilegeAfterRestart) { + PolicyItem item; + item.first.emplace_back("testuser"); + item.second.emplace_back(ActionPB::ALTER); + + AuthorizationPolicy policy; + policy.databases.emplace_back("foo"); + policy.tables.emplace_back("bar"); + policy.items.emplace_back(std::move(item)); + policy.name = "test1"; + + ASSERT_OK(ranger_.AddPolicy(policy)); + + ASSERT_OK(ranger_.Stop()); + ASSERT_OK(ranger_.Start()); + + const string kExpectedError = "Another policy already exists for matching resource"; + + Status s = ranger_.AddPolicy(std::move(policy)); + ASSERT_TRUE(s.IsRemoteError()); + ASSERT_STR_CONTAINS(s.ToString(), kExpectedError); +} + +} // namespace ranger +} // namespace kudu diff --git a/src/kudu/ranger/mini_ranger.cc b/src/kudu/ranger/mini_ranger.cc new file mode 100644 index 0000000..3351222 --- /dev/null +++ b/src/kudu/ranger/mini_ranger.cc @@ -0,0 +1,306 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "kudu/ranger/mini_ranger.h" + +#include +#include +#include + +#include + +#include "kudu/gutil/strings/substitute.h" +#include "kudu/postgres/mini_postgres.h" +#include "kudu/ranger/mini_ranger_configs.h" +#include "kudu/ranger/ranger.pb.h" +#include "kudu/util/curl_util.h" +#include "kudu/util/easy_json.h" +#include "kudu/util/env.h" +#include "kudu/util/faststring.h" +#include "kudu/util/monotime.h" +#include "kudu/util/net/net_util.h" +#include "kudu/util/path_util.h" +#include "kudu/util/slice.h" +#include "kudu/util/stopwatch.h" +#include "kudu/util/subprocess.h" +#include "kudu/util/test_util.h" + +using std::string; +using std::unique_ptr; +using strings::Substitute; + +static constexpr int kRangerStartTimeoutMs = 60000; + +namespace kudu { +namespace ranger { + +Status MiniRanger::Start() { + RETURN_NOT_OK_PREPEND(mini_pg_.Start(), "Failed to start Postgres"); + return StartRanger(); +} + +MiniRanger::~MiniRanger() { + WARN_NOT_OK(Stop(), "Failed to stop Ranger"); +} + +Status MiniRanger::Stop() { + if (process_) { + LOG(INFO) << "Stopping Ranger..."; + RETURN_NOT_OK(process_->KillAndWait(SIGTERM)); + LOG(INFO) << "Stopped Ranger"; + process_.reset(); + } + return mini_pg_.Stop(); +} + +Status MiniRanger::InitRanger(string admin_home, bool* fresh_install) { + if (env_->FileExists(admin_home)) { + *fresh_install = false; + return Status::OK(); + } + *fresh_install = true; + + RETURN_NOT_OK(env_->CreateDir(admin_home)); + + RETURN_NOT_OK(mini_pg_.AddUser("miniranger", /*super=*/ false)); + LOG(INFO) << "Created miniranger Postgres user"; + + RETURN_NOT_OK(mini_pg_.CreateDb("ranger", "miniranger")); + LOG(INFO) << "Created ranger Postgres database"; + + return Status::OK(); +} + +Status MiniRanger::CreateConfigs(const string& fqdn) { + // Ranger listens on 2 ports: + // + // - ranger_port_ is the RPC port (REST API) that the Ranger subprocess and + // EasyCurl can talk to + // - ranger_shutdown_port is the port which Ranger listens on for a shutdown + // command. We're not using this shutdown port as we simply send a SIGTERM, + // but it's necessary to set it to a random value to avoid collisions in + // parallel testing. + RETURN_NOT_OK(GetRandomPort(&ranger_port_)); + uint16_t ranger_shutdown_port; + RETURN_NOT_OK(GetRandomPort(&ranger_shutdown_port)); + string admin_home = ranger_admin_home(); + + ranger_admin_url_ = Substitute("http://$0:$1", fqdn, ranger_port_); + + // Write config files + RETURN_NOT_OK(WriteStringToFile( + env_, GetRangerInstallProperties(bin_dir(), mini_pg_.bound_port()), + JoinPathSegments(admin_home, "install.properties"))); + + RETURN_NOT_OK(WriteStringToFile( + env_, GetRangerAdminSiteXml(ranger_port_, mini_pg_.bound_port()), + JoinPathSegments(admin_home, "ranger-admin-site.xml"))); + + RETURN_NOT_OK(WriteStringToFile( + env_, GetRangerAdminDefaultSiteXml( + JoinPathSegments(bin_dir(), "postgresql.jar"), + ranger_shutdown_port), + JoinPathSegments(admin_home, "ranger-admin-default-site.xml"))); + + RETURN_NOT_OK(WriteStringToFile(env_, GetRangerCoreSiteXml(/*secure=*/ false), + JoinPathSegments(admin_home, "core-site.xml"))); + + RETURN_NOT_OK(WriteStringToFile(env_, GetRangerLog4jProperties("info"), + JoinPathSegments(admin_home, "log4j.properties"))); + + return Status::OK(); +} + +Status MiniRanger::DbSetup(const string& admin_home, const string& ews_dir, + const string& web_app_dir) { + RETURN_NOT_OK(env_->CreateDir(ews_dir)); + RETURN_NOT_OK(env_->CreateSymLink(JoinPathSegments(ranger_home_, "ews/webapp").c_str(), + web_app_dir.c_str())); + + // Much of this encapsulates setup.sh from apache/ranger[1], excluding some of + // the system-level configs. + // + // [1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh + // + // TODO(abukor): load a db dump instead as this is very slow + Subprocess db_setup( + { "python", JoinPathSegments(ranger_home_, "db_setup.py")}); + db_setup.SetEnvVars({ + { "JAVA_HOME", java_home_ }, + { "RANGER_ADMIN_HOME", ranger_home_ }, + { "RANGER_ADMIN_CONF", admin_home }, + { "XAPOLICYMGR_DIR", admin_home }, + { "RANGER_PID_DIR_PATH", ranger_home_ }, + }); + db_setup.SetCurrentDir(admin_home); + RETURN_NOT_OK(db_setup.Start()); + return db_setup.WaitAndCheckExitCode(); +} + +Status MiniRanger::StartRanger() { + bool fresh_install; + LOG_TIMING(INFO, "starting Ranger") { + LOG(INFO) << "Starting Ranger..."; + string exe; + RETURN_NOT_OK(env_->GetExecutablePath(&exe)); + const string bin_dir = DirName(exe); + RETURN_NOT_OK(FindHomeDir("hadoop", bin_dir, &hadoop_home_)); + RETURN_NOT_OK(FindHomeDir("ranger", bin_dir, &ranger_home_)); + RETURN_NOT_OK(FindHomeDir("java", bin_dir, &java_home_)); + + const string kAdminHome = ranger_admin_home(); + const string kEwsDir = JoinPathSegments(kAdminHome, "ews"); + const string kWebAppDir = JoinPathSegments(kEwsDir, "webapp"); + const string kWebInfDir = JoinPathSegments(kWebAppDir, "WEB-INF"); + const string kClassesDir = JoinPathSegments(kWebInfDir, "classes"); + const string kConfDir = JoinPathSegments(kClassesDir, "conf"); + + RETURN_NOT_OK(InitRanger(kAdminHome, &fresh_install)); + + LOG(INFO) << "Starting Ranger out of " << kAdminHome; + + string fqdn; + RETURN_NOT_OK(GetFQDN(&fqdn)); + RETURN_NOT_OK(CreateConfigs(fqdn)); + + if (fresh_install) { + RETURN_NOT_OK(DbSetup(kAdminHome, kEwsDir, kWebAppDir)); + } + + // Encapsulates ranger-admin-services.sh[1] + // + // 1. https://github.com/apache/ranger/blob/f37f5407eee8d2627a4306a25938b151f8e2ba31/embeddedwebserver/scripts/ranger-admin-services.sh + string classpath = ranger_classpath(); + + LOG(INFO) << "Using Ranger class path: " << classpath; + + LOG(INFO) << "Using FQDN: " << fqdn; + process_.reset(new Subprocess({ + JoinPathSegments(java_home_, "bin/java"), + "-Dproc_rangeradmin", + Substitute("-Dhostname=$0", fqdn), + Substitute("-Dlog4j.configuration=file:$0", + JoinPathSegments(kAdminHome, "log4j.properties")), + "-Duser=miniranger", + Substitute("-Dranger.service.host=$0", fqdn), + "-Dservername=miniranger", + Substitute("-Dcatalina.base=$0", kEwsDir), + Substitute("-Dlogdir=$0", JoinPathSegments(kAdminHome, "logs")), + "-Dranger.audit.solr.bootstrap.enabled=false", + "-cp", classpath, "org.apache.ranger.server.tomcat.EmbeddedServer" + })); + process_->SetEnvVars({ + { "XAPOLICYMGR_DIR", kAdminHome }, + { "XAPOLICYMGR_EWS_DIR", kEwsDir }, + { "RANGER_JAAS_LIB_DIR", JoinPathSegments(kWebAppDir, "ranger_jaas") }, + { "RANGER_JAAS_CONF_DIR", JoinPathSegments(kConfDir, "ranger_jaas") }, + { "JAVA_HOME", java_home_ }, + { "RANGER_PID_DIR_PATH", JoinPathSegments(data_root_, "tmppid") }, + { "RANGER_ADMIN_PID_NAME", "rangeradmin.pid" }, + { "RANGER_USER", "miniranger" }, + }); + RETURN_NOT_OK(process_->Start()); + const string ip = "127.0.0.1"; + uint16_t port; + RETURN_NOT_OK(WaitForTcpBind(process_->pid(), &port, ip, + MonoDelta::FromMilliseconds(kRangerStartTimeoutMs))); + LOG(INFO) << "Ranger bound to " << port; + LOG(INFO) << "Ranger admin URL: " << ranger_admin_url_; + } + if (fresh_install) { + RETURN_NOT_OK(CreateKuduService()); + } + + return Status::OK(); +} + +Status MiniRanger::CreateKuduService() { + EasyJson service; + service.Set("name", "kudu"); + service.Set("type", "kudu"); + + RETURN_NOT_OK_PREPEND(PostToRanger("service/plugins/services", std::move(service)), + "Failed to create Kudu service"); + LOG(INFO) << "Created Kudu service"; + return Status::OK(); +} + +Status MiniRanger::AddPolicy(AuthorizationPolicy policy) { + EasyJson policy_json; + policy_json.Set("service", "kudu"); + policy_json.Set("name", policy.name); + policy_json.Set("isEnabled", true); + + EasyJson resources = policy_json.Set("resources", EasyJson::kObject); + + if (!policy.databases.empty()) { + EasyJson databases = resources.Set("database", EasyJson::kObject); + EasyJson database_values = databases.Set("values", EasyJson::kArray); + for (const string& database : policy.databases) { + database_values.PushBack(database); + } + } + + if (!policy.tables.empty()) { + EasyJson tables = resources.Set("table", EasyJson::kObject); + EasyJson table_values = tables.Set("values", EasyJson::kArray); + for (const string& table : policy.tables) { + table_values.PushBack(table); + } + } + + if (!policy.columns.empty()) { + EasyJson columns = resources.Set("column", EasyJson::kArray); + EasyJson column_values = columns.Set("values", EasyJson::kArray); + for (const string& column : policy.columns) { + column_values.PushBack(column); + } + } + + EasyJson policy_items = policy_json.Set("policyItems", EasyJson::kArray); + for (const auto& policy_item : policy.items) { + EasyJson item = policy_items.PushBack(EasyJson::kObject); + + EasyJson users = item.Set("users", EasyJson::kArray); + for (const string& user : policy_item.first) { + users.PushBack(user); + } + + EasyJson accesses = item.Set("accesses", EasyJson::kArray); + for (const ActionPB& action : policy_item.second) { + EasyJson access = accesses.PushBack(EasyJson::kObject); + access.Set("type", ActionPB_Name(action)); + access.Set("isAllowed", true); + } + } + + RETURN_NOT_OK_PREPEND(PostToRanger("service/plugins/policies", std::move(policy_json)), + "Failed to add policy"); + return Status::OK(); +} + +Status MiniRanger::PostToRanger(string url, EasyJson payload) { + faststring result; + RETURN_NOT_OK_PREPEND(curl_.PostToURL(JoinPathSegments(ranger_admin_url_, std::move(url)), + std::move(payload.ToString()), &result, + {"Content-Type: application/json"}), + Substitute("Error recceived from Ranger: $0", result.ToString())); + return Status::OK(); +} + +} // namespace ranger +} // namespace kudu diff --git a/src/kudu/ranger/mini_ranger.h b/src/kudu/ranger/mini_ranger.h new file mode 100644 index 0000000..ff8947d --- /dev/null +++ b/src/kudu/ranger/mini_ranger.h @@ -0,0 +1,154 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include +#include +#include + +#include + +#include "kudu/gutil/port.h" +#include "kudu/gutil/strings/substitute.h" +#include "kudu/postgres/mini_postgres.h" +#include "kudu/ranger/ranger.pb.h" +#include "kudu/util/curl_util.h" +#include "kudu/util/env.h" +#include "kudu/util/path_util.h" +#include "kudu/util/status.h" +#include "kudu/util/test_util.h" + +namespace kudu { +class EasyJson; +class Subprocess; + +namespace ranger { + +// List of usernames to be used in PolicyItem; +typedef std::vector UserList; + +// Pair of a vector of usernames and a vector of allowed actions to be used in +// AuthorizationPolicy. Number of users and actions doesn't have to match, their +// cross-product is taken. +typedef std::pair> PolicyItem; + +// The AuthorizationPolicy contains a set of privileges on a resource to one or +// more users. 'items' is a vector of user-list of actions pair. This struct can +// be used to create new Ranger policies in tests. +struct AuthorizationPolicy { + std::string name; + std::vector databases; + std::vector tables; + std::vector columns; + std::vector items; +}; + +// Wrapper around Apache Ranger to be used in integration tests. +class MiniRanger { + public: + MiniRanger() + : MiniRanger(GetTestDataDirectory()) {} + + ~MiniRanger(); + + explicit MiniRanger(std::string data_root) + : data_root_(std::move(data_root)), + mini_pg_(data_root_), + env_(Env::Default()) { + curl_.set_auth(CurlAuthType::BASIC, "admin", "admin"); + } + + // Starts Ranger and its dependencies. + Status Start() WARN_UNUSED_RESULT; + + // Stops Ranger and its dependencies. + Status Stop() WARN_UNUSED_RESULT; + + // Adds a new policy to Ranger. + Status AddPolicy(AuthorizationPolicy policy) WARN_UNUSED_RESULT; + + private: + // Starts the Ranger service. + Status StartRanger() WARN_UNUSED_RESULT; + + // Initializes Ranger within 'admin_home' (home directory of the Ranger + // admin). Sets 'fresh_install' to true if 'admin_home' didn't exist before + // calling InitRanger(). + Status InitRanger(std::string admin_home, bool* fresh_install) + WARN_UNUSED_RESULT; + + // Creates configuration files. + Status CreateConfigs(const std::string& fqdn) WARN_UNUSED_RESULT; + + // Initializes Ranger's database. + Status DbSetup(const std::string& admin_home, const std::string& ews_dir, + const std::string& web_app_dir) WARN_UNUSED_RESULT; + + // Creates a Kudu service in Ranger. + Status CreateKuduService() WARN_UNUSED_RESULT; + + // Sends a POST request to Ranger with 'payload'. + Status PostToRanger(std::string url, EasyJson payload) WARN_UNUSED_RESULT; + + // Returns Ranger admin's home directory. + std::string ranger_admin_home() const { + return JoinPathSegments(data_root_, "ranger-admin"); + } + + std::string bin_dir() const { + std::string exe; + CHECK_OK(env_->GetExecutablePath(&exe)); + return DirName(exe); + } + + // Returns classpath for Ranger. + std::string ranger_classpath() const { + std::string admin_home = ranger_admin_home(); + return strings::Substitute( + "$0:$1/lib/*:$2/lib/*:$3/*:$4:$5", + admin_home, JoinPathSegments(ranger_home_, "ews"), java_home_, + hadoop_home_, JoinPathSegments(bin_dir(), "postgresql.jar"), + JoinPathSegments(ranger_home_, "ews/webapp")); + } + + // Directory in which to put all our stuff. + const std::string data_root_; + + postgres::MiniPostgres mini_pg_; + std::unique_ptr process_; + + // URL of the Ranger admin REST API. + std::string ranger_admin_url_; + + // Locations in which to find Hadoop, Ranger, and Java. + // These may be in the thirdparty build, or may be shared across tests. As + // such, their contents should be treated as read-only. + std::string hadoop_home_; + std::string ranger_home_; + std::string java_home_; + + Env* env_; + EasyCurl curl_; + + uint16_t ranger_port_; +}; + +} // namespace ranger +} // namespace kudu diff --git a/src/kudu/ranger/mini_ranger_configs.h b/src/kudu/ranger/mini_ranger_configs.h new file mode 100644 index 0000000..1998a1c --- /dev/null +++ b/src/kudu/ranger/mini_ranger_configs.h @@ -0,0 +1,349 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include + +#include "kudu/gutil/strings/substitute.h" + +namespace kudu { +namespace ranger { + +// Taken and modified from: +// https://github.com/apache/ranger/blob/master/security-admin/scripts/install.properties +// +// $0: directory containing postgresql.jar +// $1: postgres port +static const char* kInstallProperties = R"( +DB_FLAVOR=POSTGRES +SQL_CONNECTOR_JAR=$0/postgresql.jar +db_root_user=postgres +db_host=localhost:$1 +db_root_password= +db_ssl_enabled=false +db_ssl_required=false +db_ssl_verifyServerCertificate=false +db_name=ranger +db_user=miniranger +db_password= +rangerAdmin_password=admin +rangerTagsync_password=admin +rangerUsersync_password=admin +keyadmin_password=admin +mysql_core_file=db/mysql/optimized/current/ranger_core_db_mysql.sql +oracle_core_file=db/oracle/optimized/current/ranger_core_db_oracle.sql +postgres_core_file=db/postgres/optimized/current/ranger_core_db_postgres.sql +sqlserver_core_file=db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql +sqlanywhere_core_file=db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql +)"; + +// For port info, see: +// https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_reference/content/ranger-ports.html +// +// postgres DB hardcoded as "ranger" +// ranger jdbc user: miniranger +// ranger jdbc pw: miniranger +// hardcoded auth NONE +// +// $0: postgres port +// $1: admin port/RPC (REST API) port +const char* kRangerAdminSiteTemplate = R"( + + + + + + ranger.jpa.jdbc.driver + org.postgresql.Driver + + + + ranger.jpa.jdbc.url + jdbc:postgresql://localhost:$0/ranger + + + + ranger.jpa.jdbc.user + miniranger + + + + ranger.jpa.jdbc.password + miniranger + + + + + + + ranger.externalurl + http://localhost:$1 + + + + ranger.service.http.enabled + true + + + + ranger.authentication.method + NONE + + + + ranger.service.host + localhost + + + ranger.service.http.port + $1 + + + ranger.admin.cookie.name + RANGERADMINSESSIONID + + +)"; + +// ranger-admin-default-site.xml +// - postgres JDBC driver path +// - RANGER_HOME (needed for jceks/KMS), impala says this is ranger-home, but the +// conf/jcsks directory doesn't exist for us. +// +// $0: postgres JDBC driver path +// $1: ranger shutdown port +const char* kRangerAdminDefaultSiteTemplate = R"( + + + + + ranger.jdbc.sqlconnectorjar + $0 + + + + ranger.service.shutdown.port + $1 + + + + + + ranger.jpa.showsql + false + + + + ranger.jpa.jdbc.dialect + org.eclipse.persistence.platform.database.PostgreSQLPlatform + + + + ranger.jpa.jdbc.maxpoolsize + 40 + + + + + ranger.jpa.jdbc.minpoolsize + 5 + + + + + ranger.jpa.jdbc.initialpoolsize + 5 + + + + + ranger.jpa.jdbc.maxidletime + 300 + + + + + ranger.jpa.jdbc.maxstatements + 500 + + + + + ranger.jpa.jdbc.preferredtestquery + select 1; + + + + + ranger.jpa.jdbc.idleconnectiontestperiod + 60 + + + + + ranger.jpa.jdbc.credential.alias + ranger.db.password + + + + + ranger.jpa.audit.jdbc.dialect + org.eclipse.persistence.platform.database.PostgreSQLPlatform + + + + + ranger.jpa.audit.jdbc.credential.alias + ranger.auditdb.password + + + + + + ranger.jpa.audit.jdbc.driver + org.postgresql.Driver + + + + ranger.jpa.audit.jdbc.url + jdbc:log4jdbc:mysql://localhost/rangeraudit + + + + ranger.jpa.audit.jdbc.user + rangerlogger + + + + ranger.jpa.audit.jdbc.password + rangerlogger + + + +)"; + +// log4j.properties file. +// +// This is the default log4j.properties with the only difference that rootLogger +// is made configurable if it's needed for debugging. +// +// $0: log level +const char *kLog4jPropertiesTemplate = R"( +log4j.rootLogger = $0,xa_log_appender + + +# xa_logger +log4j.appender.xa_log_appender=org.apache.log4j.DailyRollingFileAppender +log4j.appender.xa_log_appender.file=$${logdir}/ranger-admin-$${hostname}-$${user}.log +log4j.appender.xa_log_appender.datePattern='.'yyyy-MM-dd +log4j.appender.xa_log_appender.append=true +log4j.appender.xa_log_appender.layout=org.apache.log4j.PatternLayout +log4j.appender.xa_log_appender.layout.ConversionPattern=%d [%t] %-5p %C{6} (%F:%L) - %m%n +# xa_log_appender : category and additivity +log4j.category.org.springframework=warn,xa_log_appender +log4j.additivity.org.springframework=false + +log4j.category.org.apache.ranger=info,xa_log_appender +log4j.additivity.org.apache.ranger=false + +log4j.category.xa=info,xa_log_appender +log4j.additivity.xa=false + +# perf_logger +log4j.appender.perf_appender=org.apache.log4j.DailyRollingFileAppender +log4j.appender.perf_appender.file=$${logdir}/ranger_admin_perf.log +log4j.appender.perf_appender.datePattern='.'yyyy-MM-dd +log4j.appender.perf_appender.append=true +log4j.appender.perf_appender.layout=org.apache.log4j.PatternLayout +log4j.appender.perf_appender.layout.ConversionPattern=%d [%t] %m%n + + +# sql_appender +log4j.appender.sql_appender=org.apache.log4j.DailyRollingFileAppender +log4j.appender.sql_appender.file=$${logdir}/ranger_admin_sql.log +log4j.appender.sql_appender.datePattern='.'yyyy-MM-dd +log4j.appender.sql_appender.append=true +log4j.appender.sql_appender.layout=org.apache.log4j.PatternLayout +log4j.appender.sql_appender.layout.ConversionPattern=%d [%t] %-5p %C{6} (%F:%L) - %m%n + +# sql_appender : category and additivity +log4j.category.org.hibernate.SQL=warn,sql_appender +log4j.additivity.org.hibernate.SQL=false + +log4j.category.jdbc.sqlonly=fatal,sql_appender +log4j.additivity.jdbc.sqlonly=false + +log4j.category.jdbc.sqltiming=warn,sql_appender +log4j.additivity.jdbc.sqltiming=false + +log4j.category.jdbc.audit=fatal,sql_appender +log4j.additivity.jdbc.audit=false + +log4j.category.jdbc.resultset=fatal,sql_appender +log4j.additivity.jdbc.resultset=false + +log4j.category.jdbc.connection=fatal,sql_appender +log4j.additivity.jdbc.connection=false +)"; + +// core-site.xml containing authentication method. +// +// $0: authn method (simple or kerberos) +const char* kCoreSiteTemplate = R"( + + + hadoop.security.authentication + $0 + + +)"; + +// Gets the contents of the install.properties file used by the db_setup.py +// script. +std::string GetRangerInstallProperties(std::string bin_dir, uint16_t pg_port) { + return strings::Substitute(kInstallProperties, bin_dir, pg_port); +} + +// Gets the contents of the ranger-admin-site.xml config that has most of the +// configuration needed to start Ranger. +std::string GetRangerAdminSiteXml(uint16_t port, uint16_t pg_port) { + return strings::Substitute(kRangerAdminSiteTemplate, pg_port, port); +} + +// Gets the ranger-admin-default-site.xml that has some additional configuration +// needed to start Ranger. It's unclear why this has to be a separate file. +std::string GetRangerAdminDefaultSiteXml(std::string pg_driver, + uint16_t shutdown_port) { + return strings::Substitute(kRangerAdminDefaultSiteTemplate, pg_driver, + shutdown_port); +} + +// Gets the contents of the log4j.properties file which is used to set up the +// logging in Ranger. The only modification to the default log4j.properties is +// the configurable log level. +std::string GetRangerLog4jProperties(std::string log_level) { + return strings::Substitute(kLog4jPropertiesTemplate, log_level); +} + +// Gets the core-site.xml that configures authentication. +std::string GetRangerCoreSiteXml(bool secure) { + return strings::Substitute(kCoreSiteTemplate, secure ? "kerberos" : "simple"); +} + +} // namespace ranger +} // namespace kudu diff --git a/src/kudu/security/test/test_pass.h b/src/kudu/security/test/test_pass.h index 5f730fe..6fa790d 100644 --- a/src/kudu/security/test/test_pass.h +++ b/src/kudu/security/test/test_pass.h @@ -26,9 +26,10 @@ namespace security { // Username and password for HTTP authentication, corresponding to // .htpasswd created by CreateTestHTPasswd() -const std::string kTestAuthString = "test:test"; +const std::string kTestAuthUsername = "test"; +const std::string kTestAuthPassword = "test"; -// Creates .htpasswd for HTTP basic authentication in the format +// Creates .htpasswd for HTTP digest authentication in the format // of 'user:realm:digest', returning the path in '*passwd_file'. Status CreateTestHTPasswd(const std::string &dir, std::string *passwd_file); diff --git a/src/kudu/server/webserver-test.cc b/src/kudu/server/webserver-test.cc index 9e02cdf..a84510e 100644 --- a/src/kudu/server/webserver-test.cc +++ b/src/kudu/server/webserver-test.cc @@ -25,7 +25,6 @@ #include #include -#include #include #include @@ -153,9 +152,9 @@ TEST_F(PasswdWebserverTest, TestPasswdMissing) { } TEST_F(PasswdWebserverTest, TestPasswdPresent) { - string auth_url = Substitute("http://$0@$1/", security::kTestAuthString, - addr_.ToString()); - ASSERT_OK(curl_.FetchURL(auth_url, &buf_)); + ASSERT_OK(curl_.set_auth(CurlAuthType::DIGEST, security::kTestAuthUsername, + security::kTestAuthPassword)); + ASSERT_OK(curl_.FetchURL(addr_.ToString(), &buf_)); } @@ -177,7 +176,7 @@ class SpnegoWebserverTest : public WebserverTest { } Status DoSpnegoCurl() { - curl_.set_use_spnego(true); + curl_.set_auth(CurlAuthType::SPNEGO); if (VLOG_IS_ON(1)) { curl_.set_verbose(true); } diff --git a/src/kudu/tools/tool.proto b/src/kudu/tools/tool.proto index c5a0bf0..107d86b 100644 --- a/src/kudu/tools/tool.proto +++ b/src/kudu/tools/tool.proto @@ -42,6 +42,9 @@ message CreateClusterRequestPB { // Whether or not the cluster should be Kerberized. optional bool enable_kerberos = 3; + // Whether or not Ranger should be enabled + optional bool enable_ranger = 9; + // Whether or not to create a Hive Metastore, and/or enable Kudu Hive // Metastore integration. optional HmsMode hms_mode = 7; diff --git a/src/kudu/tools/tool_action.h b/src/kudu/tools/tool_action.h index b8fea5a..8c78fb2 100644 --- a/src/kudu/tools/tool_action.h +++ b/src/kudu/tools/tool_action.h @@ -21,6 +21,7 @@ #include #include #include +#include #include #include diff --git a/src/kudu/tools/tool_action_test.cc b/src/kudu/tools/tool_action_test.cc index e775e9c..c2ed9c9 100644 --- a/src/kudu/tools/tool_action_test.cc +++ b/src/kudu/tools/tool_action_test.cc @@ -19,15 +19,16 @@ #include +#include #include #include #include #include -#include #include #include #include +#include #include #include #include @@ -159,6 +160,7 @@ Status ProcessRequest(const ControlShellRequestPB& req, } opts.enable_kerberos = cc.enable_kerberos(); opts.hms_mode = cc.hms_mode(); + opts.enable_ranger = cc.enable_ranger(); if (cc.has_cluster_root()) { opts.cluster_root = cc.cluster_root(); } else { diff --git a/src/kudu/util/curl_util.cc b/src/kudu/util/curl_util.cc index e7992f7..8e8425e 100644 --- a/src/kudu/util/curl_util.cc +++ b/src/kudu/util/curl_util.cc @@ -104,8 +104,9 @@ Status EasyCurl::FetchURL(const string& url, faststring* dst, Status EasyCurl::PostToURL(const string& url, const string& post_data, - faststring* dst) { - return DoRequest(url, &post_data, dst); + faststring* dst, + const vector& headers) { + return DoRequest(url, &post_data, dst, headers); } Status EasyCurl::DoRequest(const string& url, @@ -122,12 +123,29 @@ Status EasyCurl::DoRequest(const string& url, CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_SSL_VERIFYPEER, 0)); } - if (use_spnego_) { - CURL_RETURN_NOT_OK(curl_easy_setopt( - curl_, CURLOPT_HTTPAUTH, CURLAUTH_NEGOTIATE)); - // It's necessary to pass an empty user/password to trigger the authentication - // code paths in curl, even though SPNEGO doesn't use them. - CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_USERPWD, ":")); + switch (auth_type_) { + case CurlAuthType::SPNEGO: + CURL_RETURN_NOT_OK(curl_easy_setopt( + curl_, CURLOPT_HTTPAUTH, CURLAUTH_NEGOTIATE)); + break; + case CurlAuthType::DIGEST: + CURL_RETURN_NOT_OK(curl_easy_setopt( + curl_, CURLOPT_HTTPAUTH, CURLAUTH_DIGEST)); + break; + case CurlAuthType::BASIC: + CURL_RETURN_NOT_OK(curl_easy_setopt( + curl_, CURLOPT_HTTPAUTH, CURLAUTH_BASIC)); + break; + case CurlAuthType::NONE: + break; + default: + CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_HTTPAUTH, CURLAUTH_ANY)); + break; + } + + if (auth_type_ != CurlAuthType::NONE) { + CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_USERNAME, username_.c_str())); + CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_PASSWORD, password_.c_str())); } if (verbose_) { @@ -172,7 +190,6 @@ Status EasyCurl::DoRequest(const string& url, CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_NOPROXY, noproxy_.c_str())); } - CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_HTTPAUTH, CURLAUTH_ANY)); if (timeout_.Initialized()) { CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_NOSIGNAL, 1)); CURL_RETURN_NOT_OK(curl_easy_setopt( @@ -184,7 +201,7 @@ Status EasyCurl::DoRequest(const string& url, num_connects_ = static_cast(val); CURL_RETURN_NOT_OK(curl_easy_getinfo(curl_, CURLINFO_RESPONSE_CODE, &val)); - if (val != 200) { + if (val < 200 || val >= 300) { return Status::RemoteError(Substitute("HTTP $0", val)); } return Status::OK(); diff --git a/src/kudu/util/curl_util.h b/src/kudu/util/curl_util.h index c05ce66..ed01e17 100644 --- a/src/kudu/util/curl_util.h +++ b/src/kudu/util/curl_util.h @@ -32,6 +32,13 @@ namespace kudu { class faststring; +enum class CurlAuthType { + NONE, + BASIC, + DIGEST, + SPNEGO, +}; + // Simple wrapper around curl's "easy" interface, allowing the user to // fetch web pages into memory using a blocking API. // @@ -51,9 +58,12 @@ class EasyCurl { // Issue an HTTP POST to the given URL with the given data. // Returns results in 'dst' as above. + // The optional param 'headers' holds additional headers. + // e.g. {"Accept-Encoding: gzip"} Status PostToURL(const std::string& url, const std::string& post_data, - faststring* dst); + faststring* dst, + const std::vector& headers = {}); // Set whether to verify the server's SSL certificate in the case of an HTTPS // connection. @@ -69,8 +79,12 @@ class EasyCurl { timeout_ = t; } - void set_use_spnego(bool use_spnego) { - use_spnego_ = use_spnego; + Status set_auth(CurlAuthType auth_type, std::string username = "", std::string password = "") { + auth_type_ = std::move(auth_type); + username_ = std::move(username); + password_ = std::move(password); + + return Status::OK(); } // Enable verbose mode for curl. This dumps debugging output to stderr, so @@ -132,8 +146,6 @@ class EasyCurl { // Whether to return the HTTP headers with the response. bool return_headers_ = false; - bool use_spnego_ = false; - bool verbose_ = false; // The default setting for CURLOPT_FAILONERROR in libcurl is 0 (false). @@ -145,6 +157,12 @@ class EasyCurl { char errbuf_[kErrBufSize]; + std::string username_; + + std::string password_; + + CurlAuthType auth_type_ = CurlAuthType::NONE; + DISALLOW_COPY_AND_ASSIGN(EasyCurl); }; diff --git a/src/kudu/util/env-test.cc b/src/kudu/util/env-test.cc index 49556ef..3c77443 100644 --- a/src/kudu/util/env-test.cc +++ b/src/kudu/util/env-test.cc @@ -1194,4 +1194,19 @@ TEST_F(TestEnv, TestInjectEIO) { ASSERT_OK(rw2->Close()); } +TEST_F(TestEnv, TestCreateSymlink) { + const string kSrc = JoinPathSegments(test_dir_, "foo"); + const string kDst = JoinPathSegments(test_dir_, "bar"); + ASSERT_OK(env_->CreateDir(kSrc)); + ASSERT_OK(env_->CreateSymLink(kSrc, kDst)); + + unique_ptr file; + ASSERT_OK(env_->NewWritableFile(WritableFileOptions(), + JoinPathSegments(kSrc, "foobar"), + &file)); + + ASSERT_TRUE(env_->FileExists(JoinPathSegments(kDst, "foobar"))); +} + + } // namespace kudu diff --git a/src/kudu/util/env.h b/src/kudu/util/env.h index 90fb7aa..8d818e9 100644 --- a/src/kudu/util/env.h +++ b/src/kudu/util/env.h @@ -361,6 +361,9 @@ class Env { // On success, 'result' contains the answer. On failure, 'result' is unset. virtual Status IsFileWorldReadable(const std::string& path, bool* result) = 0; + // Creates symlink 'dst' that points to 'source'. + virtual Status CreateSymLink(const std::string& src, const std::string& dst) = 0; + // Special string injected into file-growing operations' random failures // (if enabled). // diff --git a/src/kudu/util/env_posix.cc b/src/kudu/util/env_posix.cc index 6545dfb..e820d44 100644 --- a/src/kudu/util/env_posix.cc +++ b/src/kudu/util/env_posix.cc @@ -1757,6 +1757,17 @@ class PosixEnv : public Env { return Status::OK(); } + virtual Status CreateSymLink(const string& src, const string& dst) override { + ThreadRestrictions::AssertIOAllowed(); + TRACE_EVENT2("io", "PosixEnv::CreateSymLink", "src", src, "dst", dst); + MAYBE_RETURN_EIO(dst, IOError(Env::kInjectedFailureStatusMsg, EIO)); + Status result; + if (symlink(src.c_str(), dst.c_str()) != 0) { + result = IOError(dst, errno); + } + return result; + } + private: // unique_ptr Deleter implementation for fts_close struct FtsCloser { diff --git a/thirdparty/LICENSE.txt b/thirdparty/LICENSE.txt index d82455d..190941b 100644 --- a/thirdparty/LICENSE.txt +++ b/thirdparty/LICENSE.txt @@ -666,3 +666,8 @@ NOTE: build-time dependency thirdparty/src/postgresql-*/: PostgreSQL license Source: https://postgresql.org NOTE: build-time dependency + +-------------------------------------------------------------------------------- +thirdparty/src/ranger-*/: Apache 2.0 license +Source: https://ranger.apache.org +NOTE: build-time dependency diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh index ad8b7bc..6f3ab98 100755 --- a/thirdparty/build-thirdparty.sh +++ b/thirdparty/build-thirdparty.sh @@ -105,6 +105,7 @@ else "gumbo-query") F_GUMBO_QUERY=1 ;; "postgres") F_POSTGRES=1 ;; "psql-jdbc") F_POSTGRES_JDBC=1 ;; + "ranger") F_RANGER=1 ;; *) echo "Unknown module: $arg"; exit 1 ;; esac done @@ -284,6 +285,19 @@ if [ -n "$F_COMMON" -o -n "$F_SENTRY" ]; then ln -nsf $SENTRY_SOURCE $PREFIX/opt/sentry fi +if [ -n "$F_COMMON" -o -n "$F_RANGER" ]; then + mkdir -p $PREFIX/opt + # Remove any hadoop jars included in the Ranger package to avoid unexpected + # runtime behavior due to different versions of hadoop jars, similarly to + # Sentry. + rm -rf $RANGER_SOURCE/lib/hadoop-[a-z-]*.jar + ln -nsf $RANGER_SOURCE $PREFIX/opt/ranger + + # Symlink conf.dist to conf + ln -nsf $PREFIX/opt/ranger/ews/webapp/WEB-INF/classes/conf.dist \ + $PREFIX/opt/ranger/ews/webapp/WEB-INF/classes/conf +fi + ### Build C dependencies without instrumentation PREFIX=$PREFIX_DEPS diff --git a/thirdparty/download-thirdparty.sh b/thirdparty/download-thirdparty.sh index 2672e03..45f8649 100755 --- a/thirdparty/download-thirdparty.sh +++ b/thirdparty/download-thirdparty.sh @@ -460,5 +460,13 @@ fetch_and_patch \ $POSTGRES_JDBC_SOURCE \ $POSTGRES_JDBC_PATCHLEVEL +RANGER_PATCHLEVEL=2 +fetch_and_patch \ + $RANGER_NAME.tar.gz \ + $RANGER_SOURCE \ + $RANGER_PATCHLEVEL \ + "patch -p1 < $TP_DIR/patches/ranger-python3.patch" \ + "patch -p0 < $TP_DIR/patches/ranger-fixscripts.patch" + echo "---------------" echo "Thirdparty dependencies downloaded successfully" diff --git a/thirdparty/patches/ranger-fixscripts.patch b/thirdparty/patches/ranger-fixscripts.patch new file mode 100644 index 0000000..d11407a --- /dev/null +++ b/thirdparty/patches/ranger-fixscripts.patch @@ -0,0 +1,14 @@ +*** ews/ranger-admin-services.sh.orig 2020-03-18 00:27:04.000000000 +0100 +--- ews/ranger-admin-services.sh 2020-03-18 00:27:14.000000000 +0100 +*************** action=$1 +*** 25,33 **** + arg2=$2 + arg3=$3 + action=`echo $action | tr '[:lower:]' '[:upper:]'` +- realScriptPath=`readlink -f $0` +- realScriptDir=`dirname $realScriptPath` +- XAPOLICYMGR_DIR=`(cd $realScriptDir/..; pwd)` + + XAPOLICYMGR_EWS_DIR=${XAPOLICYMGR_DIR}/ews + RANGER_JAAS_LIB_DIR="${XAPOLICYMGR_EWS_DIR}/ranger_jaas" +--- 25,30 ---- diff --git a/thirdparty/patches/ranger-python3.patch b/thirdparty/patches/ranger-python3.patch new file mode 100644 index 0000000..4d89156 --- /dev/null +++ b/thirdparty/patches/ranger-python3.patch @@ -0,0 +1,667 @@ +--- ./update_property.py (original) ++++ ./update_property.py (refactored) +@@ -23,7 +23,7 @@ + try: + xml = ET.parse(xml_path) + except ExpatError: +- print "Error while parsing file:"+xml_path ++ print("Error while parsing file:"+xml_path) + return -1 + root = xml.getroot() + for child in root.findall('property'): +--- ./deleteUserGroupUtil.py (original) ++++ ./deleteUserGroupUtil.py (refactored) +@@ -17,7 +17,7 @@ + import pycurl + import getpass + import logging +-from StringIO import StringIO as BytesIO ++from io import StringIO as BytesIO + def log(msg,type): + if type == 'info': + logging.info(" %s",msg) +@@ -85,8 +85,8 @@ + header.close() + c.close() + if isDebug ==True or (response_code!=200 and response_code!=204): +- print 'Request URL = ' + str(url) +- print 'Response = ' + str(headerResponse) ++ print('Request URL = ' + str(url)) ++ print('Response = ' + str(headerResponse)) + return response_code + def validateArgs(argv): + if(len(argv)<7): +@@ -217,8 +217,8 @@ + response_code=0 + try: + response_code=processRequest(url,usernamepassword,None,'get',isHttps,certfile,False) +- except pycurl.error, e: +- print e ++ except pycurl.error as e: ++ print(e) + sys.exit(1) + if response_code == 302 or response_code==401 or response_code==403: + log("[E] Authentication Error:Please try with valid credentials!","error") +--- ./dba_script.py (original) ++++ ./dba_script.py (refactored) +@@ -101,10 +101,10 @@ + f.write(msg+"\n") + f.close() + else: +- print("Unable to open file "+logFileName+" in write mode, Check file permissions.") ++ print(("Unable to open file "+logFileName+" in write mode, Check file permissions.")) + sys.exit() + else: +- print(logFileName+" is Invalid input file name! Provide valid file path to write DBA scripts:") ++ print((logFileName+" is Invalid input file name! Provide valid file path to write DBA scripts:")) + sys.exit() + else: + print("Invalid input! Provide file path to write DBA scripts:") +@@ -1494,7 +1494,7 @@ + else : + while os.path.isfile(JAVA_BIN) == False: + log("Enter java executable path: :","info") +- JAVA_BIN=raw_input() ++ JAVA_BIN=input() + log("[I] Using Java:" + str(JAVA_BIN),"info") + + +@@ -1505,7 +1505,7 @@ + XA_DB_FLAVOR='' + while XA_DB_FLAVOR == "": + log("Enter db flavour{MYSQL|ORACLE|POSTGRES|MSSQL|SQLA} :","info") +- XA_DB_FLAVOR=raw_input() ++ XA_DB_FLAVOR=input() + AUDIT_DB_FLAVOR = XA_DB_FLAVOR + + XA_DB_FLAVOR = XA_DB_FLAVOR.upper() +@@ -1519,10 +1519,10 @@ + if not dryMode: + if XA_DB_FLAVOR == "MYSQL" or XA_DB_FLAVOR == "ORACLE" or XA_DB_FLAVOR == "POSTGRES" or XA_DB_FLAVOR == "MSSQL" or XA_DB_FLAVOR == "SQLA": + log("Enter JDBC connector file for :"+XA_DB_FLAVOR,"info") +- CONNECTOR_JAR=raw_input() ++ CONNECTOR_JAR=input() + while os.path.isfile(CONNECTOR_JAR) == False: + log("JDBC connector file "+CONNECTOR_JAR+" does not exist, Please enter connector path :","error") +- CONNECTOR_JAR=raw_input() ++ CONNECTOR_JAR=input() + else: + log("[E] ---------- NO SUCH SUPPORTED DB FLAVOUR.. ----------", "error") + sys.exit(1) +@@ -1539,7 +1539,7 @@ + xa_db_host='' + while xa_db_host == "": + log("Enter DB Host :","info") +- xa_db_host=raw_input() ++ xa_db_host=input() + audit_db_host=xa_db_host + log("[I] DB Host:" + str(xa_db_host),"info") + +@@ -1554,7 +1554,7 @@ + xa_db_root_user='' + while xa_db_root_user == "": + log("Enter db root user:","info") +- xa_db_root_user=raw_input() ++ xa_db_root_user=input() + log("Enter db root password:","info") + xa_db_root_password = getpass.getpass("Enter db root password:") + +@@ -1567,7 +1567,7 @@ + db_name = '' + while db_name == "": + log("Enter DB Name :","info") +- db_name=raw_input() ++ db_name=input() + + if (quiteMode): + db_user = globalDict['db_user'] +@@ -1578,7 +1578,7 @@ + db_user='' + while db_user == "": + log("Enter db user name:","info") +- db_user=raw_input() ++ db_user=input() + + if (quiteMode): + db_password = globalDict['db_password'] +@@ -1610,7 +1610,7 @@ + audit_db_name='' + while audit_db_name == "": + log("Enter audit db name:","info") +- audit_db_name = raw_input() ++ audit_db_name = input() + + if (quiteMode): + if 'audit_db_user' in globalDict: +@@ -1622,7 +1622,7 @@ + audit_db_user='' + while audit_db_user == "": + log("Enter audit user name:","info") +- audit_db_user = raw_input() ++ audit_db_user = input() + + if (quiteMode): + if 'audit_db_password' in globalDict: +--- ./bin/ranger_install.py (original) ++++ ./bin/ranger_install.py (refactored) +@@ -16,15 +16,15 @@ + import errno + import logging + import zipfile +-import ConfigParser +-import StringIO ++import configparser ++import io + import subprocess + import fileinput + #import MySQLdb + import zipfile + import re + import shutil +-import commands ++import subprocess + from datetime import date + import getpass + import glob +@@ -78,11 +78,11 @@ + "sts = pipe.returncode + """ + ret = subprocess.call(cmd, shell=True) +- print "------------------" +- print " cmd: " + str(cmd) ++ print("------------------") ++ print(" cmd: " + str(cmd)) + #print " output: " + output +- print " ret: " + str(ret) +- print "------------------" ++ print(" ret: " + str(ret)) ++ print("------------------") + return ret, ret + #if sts is None: + # log("sts is None!!!! Manually setting to -1. PLEASE CHECK!!!!!!!!!!!!!!","info") +@@ -380,7 +380,7 @@ + file_path = os.path.dirname(os.path.realpath(__file__)) + write_conf_to_file = os.path.join(file_path, "install_config.properties") + open(write_conf_to_file,'wb') +- for key,value in conf_dict.items(): ++ for key,value in list(conf_dict.items()): + if 'PASSWORD' in key : + #call_keystore(library_path,key,value,jceks_file_path,'create') + value='' +@@ -486,8 +486,8 @@ + + cmdStr = "\""+MYSQL_BIN+"\""+" -u root --password="+db_root_password+" -h "+MYSQL_HOST+" -s -e \"select version();\"" + status, output = getstatusoutput(cmdStr) +- print "Status: " + str(status) +- print "output: " + str(output) ++ print("Status: " + str(status)) ++ print("output: " + str(output)) + + if status == 0: + log("Checking MYSQL root password DONE", "info") +@@ -696,7 +696,7 @@ + ModConfig(xapolicymgr_properties,"xa.webapp.dir", WEBAPP_ROOT.replace('\\','/' )) + + def updatePropertyToFilePy(propertyName ,newPropertyValue ,to_file): +- ret = subprocess.call(['python', '%s\update_property.py' %os.getenv("RANGER_ADMIN_HOME"), propertyName ,newPropertyValue ,to_file]) ++ ret = subprocess.call(['python', '%s\\update_property.py' %os.getenv("RANGER_ADMIN_HOME"), propertyName ,newPropertyValue ,to_file]) + if ret == 0: + log("Updated property for :"+to_file,"info") + else: +@@ -1275,7 +1275,7 @@ + statuscode = p.returncode + return statuscode, output + else: +- print 'proper command not received for input need get or create' ++ print('proper command not received for input need get or create') + + + # Entry point to script using --service +--- ./bin/ranger_usersync.py (original) ++++ ./bin/ranger_usersync.py (refactored) +@@ -20,7 +20,7 @@ + #import ranger_install + from xml.dom.minidom import getDOMImplementation + import shutil +-import commands ++import subprocess + import re + + cmd = sys.argv[0] +@@ -117,7 +117,7 @@ + text_re = re.compile('>\n\s+([^<>\s].*?)\n\s+\g<1>\n\s+([^<>\s].*?)\n\s+\g<1> 1 and len(argv) < 8 : + for i in range(1, len(sys.argv)) : + if sys.argv[i] == "-u" : +@@ -128,13 +128,13 @@ + if userRole.lower() == "-p" or userRole.lower() == "-r" or userRole.lower() == "-u": + userRoleMsgFlag = True + if userNameMsgFlag == True or passwordMsgFlag == True or userRoleMsgFlag == True : +- print msgPrompt + " or \n" + msgCommand + "\n " +msgRoleList ++ print(msgPrompt + " or \n" + msgCommand + "\n " +msgRoleList) + if userNameMsgFlag == True : +- userName = raw_input('Enter a user name: ') ++ userName = input('Enter a user name: ') + if passwordMsgFlag == True : + password = getpass.getpass("Enter user password:") + if userRoleMsgFlag == True : +- userRole = raw_input('Enter a role: ') ++ userRole = input('Enter a role: ') + if userName != "" and password != "" : + if os_name == "LINUX": + path = os.path.join("%s","WEB-INF","classes","conf:%s","WEB-INF","classes","lib","*:%s","WEB-INF",":%s","META-INF",":%s","WEB-INF","lib","*:%s","WEB-INF","classes",":%s","WEB-INF","classes","META-INF:%s/*")%(app_home ,app_home ,app_home, app_home, app_home, app_home ,app_home,ews_lib) +--- ./restrict_permissions.py (original) ++++ ./restrict_permissions.py (refactored) +@@ -82,10 +82,10 @@ + f.write(msg+"\n") + f.close() + else: +- print("Unable to open file "+logFileName+" in write mode, Check file permissions.") ++ print(("Unable to open file "+logFileName+" in write mode, Check file permissions.")) + sys.exit() + else: +- print(logFileName+" is Invalid input file name! Provide valid file path to write DBA scripts:") ++ print((logFileName+" is Invalid input file name! Provide valid file path to write DBA scripts:")) + sys.exit() + else: + print("Invalid input! Provide file path to write DBA scripts:") +@@ -306,7 +306,7 @@ + else : + while os.path.isfile(JAVA_BIN) == False: + log("Enter java executable path: :","info") +- JAVA_BIN=raw_input() ++ JAVA_BIN=input() + log("[I] Using Java:" + str(JAVA_BIN),"info") + + if (quiteMode): +@@ -316,7 +316,7 @@ + XA_DB_FLAVOR='' + while XA_DB_FLAVOR == "": + log("Enter db flavour{MYSQL} :","info") +- XA_DB_FLAVOR=raw_input() ++ XA_DB_FLAVOR=input() + AUDIT_DB_FLAVOR = XA_DB_FLAVOR + XA_DB_FLAVOR = XA_DB_FLAVOR.upper() + AUDIT_DB_FLAVOR = AUDIT_DB_FLAVOR.upper() +@@ -328,10 +328,10 @@ + else: + if XA_DB_FLAVOR == "MYSQL": + log("Enter JDBC connector file for :"+XA_DB_FLAVOR,"info") +- CONNECTOR_JAR=raw_input() ++ CONNECTOR_JAR=input() + while os.path.isfile(CONNECTOR_JAR) == False: + log("JDBC connector file "+CONNECTOR_JAR+" does not exist, Please enter connector path :","error") +- CONNECTOR_JAR=raw_input() ++ CONNECTOR_JAR=input() + else: + log("[E] ---------- NO SUCH SUPPORTED DB FLAVOUR.. ----------", "error") + sys.exit(1) +@@ -343,7 +343,7 @@ + xa_db_host='' + while xa_db_host == "": + log("Enter DB Host :","info") +- xa_db_host=raw_input() ++ xa_db_host=input() + audit_db_host=xa_db_host + log("[I] DB Host:" + str(xa_db_host),"info") + +@@ -354,7 +354,7 @@ + xa_db_root_user='' + while xa_db_root_user == "": + log("Enter db root user:","info") +- xa_db_root_user=raw_input() ++ xa_db_root_user=input() + log("Enter db root password:","info") + xa_db_root_password = getpass.getpass("Enter db root password:") + +@@ -364,7 +364,7 @@ + db_name = '' + while db_name == "": + log("Enter DB Name :","info") +- db_name=raw_input() ++ db_name=input() + + if (quiteMode): + db_user = globalDict['db_user'] +@@ -372,7 +372,7 @@ + db_user='' + while db_user == "": + log("Enter db user name:","info") +- db_user=raw_input() ++ db_user=input() + + if (quiteMode): + db_password = globalDict['db_password'] +@@ -388,7 +388,7 @@ + audit_db_name='' + while audit_db_name == "": + log("Enter audit db name:","info") +- audit_db_name = raw_input() ++ audit_db_name = input() + + if (quiteMode): + audit_db_user = globalDict['audit_db_user'] +@@ -396,7 +396,7 @@ + audit_db_user='' + while audit_db_user == "": + log("Enter audit user name:","info") +- audit_db_user = raw_input() ++ audit_db_user = input() + + if (quiteMode): + audit_db_password = globalDict['audit_db_password'] +--- ./upgrade_admin.py (original) ++++ ./upgrade_admin.py (refactored) +@@ -13,9 +13,9 @@ + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + # See the License for the specific language governing permissions and + # limitations under the License. +-import StringIO ++import io + import xml.etree.ElementTree as ET +-import ConfigParser ++import configparser + import os,sys,getopt + from os import listdir + from os.path import isfile, join, dirname, basename +@@ -29,10 +29,10 @@ + tempLibFolder = "./upgrade-temp" + + def showUsage(): +- print "upgrade_admin.py [-g] [-h]" +- print "This script will generate %s based on currently installed ranger (v0.4.*) configuration." % (installPropFileName) +- print " -g option will generate ranger-admin-site.xml in the current directory." +- print " -h will display help text." ++ print("upgrade_admin.py [-g] [-h]") ++ print("This script will generate %s based on currently installed ranger (v0.4.*) configuration." % (installPropFileName)) ++ print(" -g option will generate ranger-admin-site.xml in the current directory.") ++ print(" -h will display help text.") + + try: + opts, args = getopt.getopt(sys.argv[1:],"gh") +@@ -126,16 +126,16 @@ + archiveDir = dirname(originalFileName) + archiveFileName = "." + basename(originalFileName) + "." + (strftime("%d%m%Y%H%M%S", localtime())) + movedFileName = join(archiveDir,archiveFileName) +- print "INFO: moving [%s] to [%s] ......." % (originalFileName,movedFileName) ++ print("INFO: moving [%s] to [%s] ......." % (originalFileName,movedFileName)) + os.rename(originalFileName, movedFileName) + + def getPropertiesConfigMap(configFileName): + ret = {} +- config = StringIO.StringIO() ++ config = io.StringIO() + config.write('[dummysection]\n') + config.write(open(configFileName).read()) + config.seek(0,os.SEEK_SET) +- fcp = ConfigParser.ConfigParser() ++ fcp = configparser.ConfigParser() + fcp.optionxform = str + fcp.readfp(config) + for k,v in fcp.items('dummysection'): +@@ -144,11 +144,11 @@ + + def getPropertiesKeyList(configFileName): + ret = [] +- config = StringIO.StringIO() ++ config = io.StringIO() + config.write('[dummysection]\n') + config.write(open(configFileName).read()) + config.seek(0,os.SEEK_SET) +- fcp = ConfigParser.ConfigParser() ++ fcp = configparser.ConfigParser() + fcp.optionxform = str + fcp.readfp(config) + for k,v in fcp.items('dummysection'): +@@ -167,10 +167,10 @@ + root = tree.getroot() + for config in root.iter('property'): + name = config.find('name').text +- if (name in prop.keys()): ++ if (name in list(prop.keys())): + config.find('value').text = prop[name] + else: +- print "ERROR: key not found: %s" % (name) ++ print("ERROR: key not found: %s" % (name)) + if isfile(xmlOutputFileName): + archiveFile(xmlOutputFileName) + tree.write(xmlOutputFileName) +@@ -192,15 +192,15 @@ + webserverConfigFileName = join(configDirectory, webserverConfigFile) + webconfig = getPropertiesConfigMap(webserverConfigFileName) + +- for k in config2xmlMAP.keys(): ++ for k in list(config2xmlMAP.keys()): + xmlKey = config2xmlMAP[k] +- if (k in xaSysProps.keys()): ++ if (k in list(xaSysProps.keys())): + xmlVal = xaSysProps[k] +- elif (k in xaLdapProps.keys()): ++ elif (k in list(xaLdapProps.keys())): + xmlVal = xaLdapProps[k] +- elif (k in unixauthProps.keys()): ++ elif (k in list(unixauthProps.keys())): + xmlVal = unixauthProps[k] +- elif (k in webconfig.keys()): ++ elif (k in list(webconfig.keys())): + xmlVal = webconfig[k] + else: + xmlVal = 'Unknown' +@@ -251,7 +251,7 @@ + installProps['db_name'] = '' + installProps['audit_db_name'] = '' + else: +- print "ERROR: Unable to determine the DB_FLAVOR from url [%]" % (jdbcUrl) ++ print("ERROR: Unable to determine the DB_FLAVOR from url [%]" % (jdbcUrl)) + sys.exit(1) + + installProps['db_user'] = xaSysProps['jdbc.user'] +@@ -296,7 +296,7 @@ + defValMap = getPropertiesConfigMap(installFileName) + + +- for wk,wv in webconfig.iteritems(): ++ for wk,wv in webconfig.items(): + nk = "ranger." + wk + nk = nk.replace('.','_') + installProps[nk] = wv +@@ -314,18 +314,18 @@ + + outf = open(outFileName, 'w') + +- print >> outf, "#" +- print >> outf, "# -----------------------------------------------------------------------------------" +- print >> outf, "# This file is generated as part of upgrade script and should be deleted after upgrade" +- print >> outf, "# Generated at %s " % (strftime("%d/%m/%Y %H:%M:%S", localtime())) +- print >> outf, "# -----------------------------------------------------------------------------------" +- print >> outf, "#" ++ print("#", file=outf) ++ print("# -----------------------------------------------------------------------------------", file=outf) ++ print("# This file is generated as part of upgrade script and should be deleted after upgrade", file=outf) ++ print("# Generated at %s " % (strftime("%d/%m/%Y %H:%M:%S", localtime())), file=outf) ++ print("# -----------------------------------------------------------------------------------", file=outf) ++ print("#", file=outf) + + for key in keyList: + if (key in props): +- print >> outf, "%s=%s" % (key,props[key]) ++ print("%s=%s" % (key,props[key]), file=outf) + else: +- print >> outf, "# Default value for [%s] is used\n%s=%s\n#---" % (key, key,defValMap[key]) ++ print("# Default value for [%s] is used\n%s=%s\n#---" % (key, key,defValMap[key]), file=outf) + + outf.flush() + outf.close() +--- ./changeusernameutil.py (original) ++++ ./changeusernameutil.py (refactored) +@@ -73,7 +73,7 @@ + else: + while os.path.isfile(JAVA_BIN) == False: + log("Enter java executable path: :","info") +- JAVA_BIN=raw_input() ++ JAVA_BIN=input() + log("[I] Using Java:" + str(JAVA_BIN),"info") + + USERNAME = '' +@@ -90,8 +90,8 @@ + sys.exit(1) + + while userName == "": +- print "Enter user name:" +- userName=raw_input() ++ print("Enter user name:") ++ userName=input() + + while oldPassword == "": + oldPassword=getpass.getpass("Enter current password:") +--- ./ranger_credential_helper.py (original) ++++ ./ranger_credential_helper.py (refactored) +@@ -20,11 +20,11 @@ + from optparse import OptionParser + + if os.getenv('JAVA_HOME') is None: +- print "ERROR: JAVA_HOME environment property was not defined, exit." ++ print("ERROR: JAVA_HOME environment property was not defined, exit.") + sys.exit(1) + else: + JAVA_BIN=os.path.join(os.getenv('JAVA_HOME'),'bin','java') +-print "Using Java:" + str(JAVA_BIN) ++print("Using Java:" + str(JAVA_BIN)) + + def main(): + +@@ -54,9 +54,9 @@ + output, error = p.communicate() + statuscode = p.returncode + if statuscode == 0: +- print "Alias " + aliasKey + " created successfully!" ++ print("Alias " + aliasKey + " created successfully!") + else : +- print "Error creating Alias!! Error: " + str(error) ++ print("Error creating Alias!! Error: " + str(error)) + + elif getorcreate == 'get': + commandtorun = [JAVA_BIN, '-cp', finalLibPath, 'org.apache.ranger.credentialapi.buildks' ,'get', aliasKey, '-provider',finalFilePath] +@@ -64,12 +64,12 @@ + output, error = p.communicate() + statuscode = p.returncode + if statuscode == 0: +- print "Alias : " + aliasKey + " Value : " + str(output) ++ print("Alias : " + aliasKey + " Value : " + str(output)) + else : +- print "Error getting value!! Error: " + str(error) ++ print("Error getting value!! Error: " + str(error)) + + else: +- print 'Invalid Arguments!!' ++ print('Invalid Arguments!!') + + if __name__ == '__main__': + main() +--- ./db_setup.py (original) ++++ ./db_setup.py (refactored) +@@ -55,7 +55,7 @@ def check_output(query): + elif os_name == "WINDOWS": + p = subprocess.Popen(query, stdout=subprocess.PIPE, shell=True) + output = p.communicate ()[0] +- return output ++ return output.decode() + + def log(msg,type): + if type == 'info': +@@ -423,7 +423,7 @@ + key3 = int(version.strip("J")) + my_dict[key3] = filename + +- keylist = my_dict.keys() ++ keylist = list(my_dict.keys()) + keylist.sort() + for key in keylist: + #print "%s: %s" % (key, my_dict[key]) +@@ -1035,20 +1035,20 @@ + lib_home = os.path.join(RANGER_ADMIN_HOME,"ews","webapp","WEB-INF","lib","*") + get_ranger_version_cmd="%s -cp %s org.apache.ranger.common.RangerVersionInfo"%(JAVA_BIN,lib_home) + ranger_version = check_output(get_ranger_version_cmd).split("\n")[1] +- except Exception, error: ++ except Exception as error: + ranger_version='' + + try: + if ranger_version=="" or ranger_version=="ranger-admin - None": + script_path = os.path.join(RANGER_ADMIN_HOME,"ews","ranger-admin-services.sh") + ranger_version=check_output(script_path +" version").split("\n")[1] +- except Exception, error: ++ except Exception as error: + ranger_version='' + + try: + if ranger_version=="" or ranger_version=="ranger-admin - None": + ranger_version=check_output("ranger-admin version").split("\n")[1] +- except Exception, error: ++ except Exception as error: + ranger_version='' + + if ranger_version=="" or ranger_version is None: +--- ./changepasswordutil.py (original) ++++ ./changepasswordutil.py (refactored) +@@ -73,7 +73,7 @@ + else: + while os.path.isfile(JAVA_BIN) == False: + log("Enter java executable path: :","info") +- JAVA_BIN=raw_input() ++ JAVA_BIN=input() + log("[I] Using Java:" + str(JAVA_BIN),"info") + + USERNAME = '' +@@ -90,8 +90,8 @@ + sys.exit(1) + + while userName == "": +- print "Enter user name:" +- userName=raw_input() ++ print("Enter user name:") ++ userName=input() + + while oldPassword == "": + oldPassword=getpass.getpass("Enter current password:") diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh index ac58bb2..f30380e 100644 --- a/thirdparty/vars.sh +++ b/thirdparty/vars.sh @@ -262,3 +262,12 @@ POSTGRES_SOURCE=$TP_SOURCE_DIR/$POSTGRES_NAME POSTGRES_JDBC_VERSION=42.2.10 POSTGRES_JDBC_NAME=postgresql-$POSTGRES_JDBC_VERSION POSTGRES_JDBC_SOURCE=$TP_SOURCE_DIR/$POSTGRES_JDBC_NAME + +# If you need to rebuild the tarball for a specific hash instead of a release, +# run the following commands: +# mvn versions:set -DnewVersion=$(git rev-parse HEAD) +# mvn versions:update-child-modules +# mvn package +RANGER_VERSION=f37f5407eee8d2627a4306a25938b151f8e2ba31 +RANGER_NAME=ranger-$RANGER_VERSION-admin +RANGER_SOURCE=$TP_SOURCE_DIR/$RANGER_NAME