Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2B5CC200B50 for ; Fri, 15 Jul 2016 02:57:51 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 29F9E160A85; Fri, 15 Jul 2016 00:57:51 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4AEAC160A63 for ; Fri, 15 Jul 2016 02:57:50 +0200 (CEST) Received: (qmail 31135 invoked by uid 500); 15 Jul 2016 00:57:49 -0000 Mailing-List: contact commits-help@kudu.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kudu.incubator.apache.org Delivered-To: mailing list commits@kudu.incubator.apache.org Received: (qmail 31126 invoked by uid 99); 15 Jul 2016 00:57:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Jul 2016 00:57:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 09482CC3D1 for ; Fri, 15 Jul 2016 00:57:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.507 X-Spam-Level: X-Spam-Status: No, score=-4.507 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.287] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id CxKF4Cd_bBmx for ; Fri, 15 Jul 2016 00:57:45 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with SMTP id D65825FE5E for ; Fri, 15 Jul 2016 00:57:44 +0000 (UTC) Received: (qmail 30956 invoked by uid 99); 15 Jul 2016 00:57:44 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Jul 2016 00:57:44 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 060CCE964E; Fri, 15 Jul 2016 00:57:43 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: mpercy@apache.org To: commits@kudu.incubator.apache.org Date: Fri, 15 Jul 2016 00:57:43 -0000 Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: [1/6] incubator-kudu git commit: Disable exactly once semantics by default and add a flag to enable it for tests archived-at: Fri, 15 Jul 2016 00:57:51 -0000 Repository: incubator-kudu Updated Branches: refs/heads/master b666cc07e -> 59ab14b9a Disable exactly once semantics by default and add a flag to enable it for tests Since exactly once semantics is still missing some pieces, like garbage collection this disables it by default on the server, but adds a flag to allow enabling it and enables it in all tablet server tests, by default. Change-Id: I77096be608afb31194f62f04a946bd3f42537a35 Reviewed-on: http://gerrit.cloudera.org:8080/3506 Reviewed-by: Jean-Daniel Cryans Tested-by: Kudu Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/257ba292 Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/257ba292 Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/257ba292 Branch: refs/heads/master Commit: 257ba2923ad962193fbca49b3d22b9fb6a578509 Parents: b666cc0 Author: David Alves Authored: Mon Jun 27 02:09:09 2016 -0700 Committer: David Ribeiro Alves Committed: Thu Jul 14 22:57:20 2016 +0000 ---------------------------------------------------------------------- .../integration-tests/external_mini_cluster-itest-base.h | 2 ++ src/kudu/integration-tests/remote_bootstrap-itest.cc | 3 +++ src/kudu/integration-tests/ts_itest-base.h | 4 ++++ src/kudu/rpc/rpc-test-base.h | 3 +++ src/kudu/rpc/service_if.cc | 10 +++++++++- src/kudu/tserver/tablet_server-test-base.h | 5 +++++ 6 files changed, 26 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/257ba292/src/kudu/integration-tests/external_mini_cluster-itest-base.h ---------------------------------------------------------------------- diff --git a/src/kudu/integration-tests/external_mini_cluster-itest-base.h b/src/kudu/integration-tests/external_mini_cluster-itest-base.h index e2a9cf5..0fde8d4 100644 --- a/src/kudu/integration-tests/external_mini_cluster-itest-base.h +++ b/src/kudu/integration-tests/external_mini_cluster-itest-base.h @@ -77,6 +77,8 @@ void ExternalMiniClusterITestBase::StartCluster(const std::vector& opts.num_tablet_servers = num_tablet_servers; opts.extra_master_flags = extra_master_flags; opts.extra_tserver_flags = extra_ts_flags; + // TODO remove when this is enabled by default. + opts.extra_tserver_flags.push_back("--enable_exactly_once"); opts.extra_tserver_flags.push_back("--never_fsync"); // fsync causes flakiness on EC2. cluster_.reset(new ExternalMiniCluster(opts)); ASSERT_OK(cluster_->Start()); http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/257ba292/src/kudu/integration-tests/remote_bootstrap-itest.cc ---------------------------------------------------------------------- diff --git a/src/kudu/integration-tests/remote_bootstrap-itest.cc b/src/kudu/integration-tests/remote_bootstrap-itest.cc index e9bf883..661ed21 100644 --- a/src/kudu/integration-tests/remote_bootstrap-itest.cc +++ b/src/kudu/integration-tests/remote_bootstrap-itest.cc @@ -112,6 +112,9 @@ void RemoteBootstrapITest::StartCluster(const vector& extra_tserver_flag ExternalMiniClusterOptions opts; opts.num_tablet_servers = num_tablet_servers; opts.extra_tserver_flags = extra_tserver_flags; + // Enable EO semantics for tests. + // TODO remove this once EO is the default. + opts.extra_tserver_flags.push_back("--enable_exactly_once"); opts.extra_tserver_flags.push_back("--never_fsync"); // fsync causes flakiness on EC2. opts.extra_master_flags = extra_master_flags; cluster_.reset(new ExternalMiniCluster(opts)); http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/257ba292/src/kudu/integration-tests/ts_itest-base.h ---------------------------------------------------------------------- diff --git a/src/kudu/integration-tests/ts_itest-base.h b/src/kudu/integration-tests/ts_itest-base.h index ad38b3e..225c942 100644 --- a/src/kudu/integration-tests/ts_itest-base.h +++ b/src/kudu/integration-tests/ts_itest-base.h @@ -97,6 +97,10 @@ class TabletServerIntegrationTestBase : public TabletServerTestBase { opts.num_tablet_servers = FLAGS_num_tablet_servers; opts.data_root = GetTestPath(data_root_path); + // Enable exactly once semantics for tests. + // TODO remove this once we have ResultTracker GC + opts.extra_tserver_flags.push_back("--enable_exactly_once"); + // If the caller passed no flags use the default ones, where we stress consensus by setting // low timeouts and frequent cache misses. if (non_default_ts_flags.empty()) { http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/257ba292/src/kudu/rpc/rpc-test-base.h ---------------------------------------------------------------------- diff --git a/src/kudu/rpc/rpc-test-base.h b/src/kudu/rpc/rpc-test-base.h index ec0edc9..7b82b08 100644 --- a/src/kudu/rpc/rpc-test-base.h +++ b/src/kudu/rpc/rpc-test-base.h @@ -45,6 +45,8 @@ #include "kudu/util/test_util.h" #include "kudu/util/trace.h" +DECLARE_bool(enable_exactly_once); + namespace kudu { namespace rpc { using kudu::rpc_test::AddRequestPB; @@ -318,6 +320,7 @@ class RpcTestBase : public KuduTest { n_server_reactor_threads_(3), keepalive_time_ms_(1000), metric_entity_(METRIC_ENTITY_server.Instantiate(&metric_registry_, "test.rpc_test")) { + FLAGS_enable_exactly_once = true; } void SetUp() override { http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/257ba292/src/kudu/rpc/service_if.cc ---------------------------------------------------------------------- diff --git a/src/kudu/rpc/service_if.cc b/src/kudu/rpc/service_if.cc index 4a8c387..8b47ce9 100644 --- a/src/kudu/rpc/service_if.cc +++ b/src/kudu/rpc/service_if.cc @@ -27,6 +27,12 @@ #include "kudu/rpc/inbound_call.h" #include "kudu/rpc/rpc_context.h" #include "kudu/rpc/rpc_header.pb.h" +#include "kudu/util/flag_tags.h" + +// TODO remove this once we have ResultTracker GC +DEFINE_bool(enable_exactly_once, false, "Whether to enable exactly once semantics on the client " + "(experimental)."); +TAG_FLAG(enable_exactly_once, experimental); using google::protobuf::Message; using std::string; @@ -92,7 +98,9 @@ void GeneratedServiceIf::Handle(InboundCall *call) { } Message* resp = method_info->resp_prototype->New(); - bool track_result = call->header().has_request_id() && method_info->track_result; + bool track_result = call->header().has_request_id() + && method_info->track_result + && FLAGS_enable_exactly_once; RpcContext* ctx = new RpcContext(call, req.release(), resp, http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/257ba292/src/kudu/tserver/tablet_server-test-base.h ---------------------------------------------------------------------- diff --git a/src/kudu/tserver/tablet_server-test-base.h b/src/kudu/tserver/tablet_server-test-base.h index 8f0b757..1d5022e 100644 --- a/src/kudu/tserver/tablet_server-test-base.h +++ b/src/kudu/tserver/tablet_server-test-base.h @@ -58,6 +58,7 @@ DEFINE_int32(rpc_timeout, 1000, "Timeout for RPC calls, in seconds"); DEFINE_int32(num_updater_threads, 1, "Number of updating threads to launch"); DECLARE_bool(log_force_fsync_all); +DECLARE_bool(enable_exactly_once); DECLARE_bool(enable_maintenance_manager); DECLARE_bool(enable_data_block_fsync); DECLARE_int32(heartbeat_rpc_timeout_ms); @@ -80,6 +81,10 @@ class TabletServerTestBase : public KuduTest { // maintenance operations at predetermined times. FLAGS_enable_maintenance_manager = false; + // Enable exactly once semantics, for tests. + // TODO remove this once we have ResultTracker GC + FLAGS_enable_exactly_once = true; + // Decrease heartbeat timeout: we keep re-trying heartbeats when a // single master server fails due to a network error. Decreasing // the hearbeat timeout to 1 second speeds up unit tests which