Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 186F32004A0 for ; Wed, 16 Aug 2017 23:32:11 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 16A33169AF2; Wed, 16 Aug 2017 21:32:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 72159169AF1 for ; Wed, 16 Aug 2017 23:32:08 +0200 (CEST) Received: (qmail 39119 invoked by uid 500); 16 Aug 2017 21:32:06 -0000 Mailing-List: contact user-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.apache.org Delivered-To: mailing list user@kudu.apache.org Received: (qmail 39109 invoked by uid 99); 16 Aug 2017 21:32:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Aug 2017 21:32:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id E7EC7C046E for ; Wed, 16 Aug 2017 21:32:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.601 X-Spam-Level: * X-Spam-Status: No, score=1.601 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id BgqQ81kwgjb4 for ; Wed, 16 Aug 2017 21:31:54 +0000 (UTC) Received: from mail-ua0-f171.google.com (mail-ua0-f171.google.com [209.85.217.171]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5CBDA60D37 for ; Wed, 16 Aug 2017 21:31:53 +0000 (UTC) Received: by mail-ua0-f171.google.com with SMTP id f9so18590835uaf.4 for ; Wed, 16 Aug 2017 14:31:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=69JHqmYAWgzmRgJgjWgNJ/zkN3bLCIKUaM2UktyhDb0=; b=FUIqSED38BN02ew+4io1+tP4+0IKlJ3YpCMz5ZR2DJf20bkmv1STq7T7rFlUjrvt5P ar61L9l0IpjbYUmE3ihAax0i1Q1ZXfFHzvsCwddkDBcRieryConlQMC/JXg51H8LRcZB NeCJgOpLUivM8KP7S1EwU6ckXsg+U/v33cC5dw8Rm5VKEqW8zWHJVk885HBKQBnC+940 LWyaH0Bo9BhVkYao5ADX1h2Bk8mzOzYu7IoYB8jIiMSxk+PKetRhTgXAzGjzqgFqw+uA JT1QxdiPDijktf/TUAjBGxR0kGiNGoUFVQjL9Lb+uKN0GlgmtaJ/0VWVueQuuwxAKgNp g9JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=69JHqmYAWgzmRgJgjWgNJ/zkN3bLCIKUaM2UktyhDb0=; b=nyzbzZt75ngF3S/qhy6rkeo54D/I1fhm7bMDAMi70+L2r2XzuLcmDZ0e2J6uoPN44a U82au8Op56C0CV4/3icAmwcHcYT2sgy9qPJi25ZKoTSLI+6xubsQdfmls7/IrGwicvpZ oQUGgm8/zWIQETJesrVl4ZMoStLwlrxvV0l5im+5hcWgS8c6sNKot5eVE8aoCZqXu+2c e22et5Y1oDvOllTDGLAb0rGKMhwxThhUzetpGCkuMzssPcUwgh3tZca6ZO566U1OvixH jObqs9JraW2/rSWQQxp3yz6iQW/PXZav0Q4Dgg5sk+B/dsFmAV9BDIx5OVn2DswAV6pv ZeaQ== X-Gm-Message-State: AHYfb5iEGtiASdqKFBDToipMKZsHtydpEXe2lezlaBxPwZ5o+9C+XGld O7IT76kLR9zCgdDg4p81JRwu9faXiGNNJMc= X-Received: by 10.159.59.19 with SMTP id i19mr2206951uah.147.1502919111368; Wed, 16 Aug 2017 14:31:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.24.200 with HTTP; Wed, 16 Aug 2017 14:31:30 -0700 (PDT) In-Reply-To: References: <41ddc240-a8e7-87b3-2765-8311db5b1329@cloudera.com> From: Todd Lipcon Date: Wed, 16 Aug 2017 14:31:30 -0700 Message-ID: Subject: Re: Configure Impala for Kudu on Separate Cluster To: user@kudu.apache.org Content-Type: multipart/alternative; boundary="f403043c3a24c2e7940556e59f53" archived-at: Wed, 16 Aug 2017 21:32:11 -0000 --f403043c3a24c2e7940556e59f53 Content-Type: text/plain; charset="UTF-8" On Wed, Aug 16, 2017 at 6:16 AM, Benjamin Kim wrote: > Hi, > > I found 2 issues. First, network connection is blocked. I filed a request > to open it. Second, Kudu 1.4 had problems registering any tablet server. > So, I reverted back to 1.3.1. The error had to do security and how IP > routing was not trusted. > Can you clarify what problem you are hitting? Is this the security improvement that prevents from running with public IP addresses without security enabled? You can override this if you're sure that your network is secured by using the --trusted-subnets option (see the security docs for details) -Todd > > Also, I installed Kudu onto another cluster where I knew the network would > not be a problem, and it worked. > > Cheers, > Ben > > On Wed, Aug 16, 2017 at 12:44 AM Alexey Serbin > wrote: > >> Ben, >> >> As Todd mentioned, it might be some network connectivity problem. I >> would suspect some issues with connectivity between the node where the >> Impala shell is running and the Kudu master node. >> >> To start troubleshooting, I would verify that the node where you run the >> Impala shell (that's 172.35.120.191, right?) can establish a TCP >> connection to the master RPC end-point. E.g., try to run from the >> command shell at 172.35.120.191: >> >> telnet 172.35.121.101 7051 >> >> Would it succeed? >> >> Also, if running multi-master Kudu cluster, it might happen that masters >> cannot communicate with each other. To troubleshoot that, I would try >> to establish a TCP connection to the RPC end-point of the master at one >> node from another master node. E.g., if using telnet, from >> , in the command-line shell: >> >> telnet 7051 >> (just substitute and with appropriate >> hostnames/IP addresses). >> >> >> >> Best regards, >> >> Alexey >> >> >> On 8/15/17 9:53 PM, Benjamin Kim wrote: >> > Todd, >> > >> > Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master >> > config (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer Kudu Master - >> > prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] Connection >> > reset on [id: 0x6232f33f, /172.35.120.191:47848 >> > :> /172.35.121.101:7051 >> > ] >> > >> > We got this error trying to use Kudu from within the cluster. Do you >> > know what this means? >> > >> > Cheers, >> > Ben >> > >> > >> > On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon > > > wrote: >> > >> > Is there a possibility that the remote node (prod-dc1-datanode151) >> > is firewalled off from whatever host you are submitting the query >> > to? The error message is admittedly pretty bad, but it basically >> > means it's getting "connection refused", indicating that either >> > there is no master running on that host or it has been blocked (eg >> > an iptables REJECT rule) >> > >> > -Todd >> > >> > On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim > > > wrote: >> > >> > Hi Todd, >> > >> > I tried to create a Kudu table using impala shell, and I got >> > this error. >> > >> > create table my_first_table >> > ( >> > id bigint, >> > name string, >> > primary key(id) >> > ) >> > partition by hash partitions 16 >> > stored as kudu; >> > Query: create table my_first_table >> > ( >> > id bigint, >> > name string, >> > primary key(id) >> > ) >> > partition by hash partitions 16 >> > stored as kudu >> > ERROR: ImpalaRuntimeException: Error creating Kudu table >> > 'impala::default.my_first_table' >> > CAUSED BY: NonRecoverableException: Too many attempts: >> > KuduRpc(method=ListTables, tablet=null, attempt=101, >> > DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms] >> > querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to >> > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [2ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [5ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [21ms] querying master, [22ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [24ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [26ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [41ms] querying master, [41ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [43ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [46ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [62ms] querying master, [62ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [64ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [66ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [81ms] querying master, [81ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [84ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [86ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [122ms] querying master, [122ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [123ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [124ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [161ms] querying master, [162ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [163ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [166ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [302ms] querying master, [302ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [304ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [306ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [382ms] querying master, [382ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [384ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [386ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [622ms] querying master, [622ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [623ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [626ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [701ms] querying master, [702ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [703ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [706ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [2062ms] querying master, [2062ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [2065ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [2068ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [4661ms] querying master, [4662ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [4663ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [4666ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [6961ms] querying master, [6962ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [6964ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [6967ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [9102ms] querying master, [9102ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [9104ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [9106ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [12001ms] querying master, [12002ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [12003ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [12006ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [15822ms] querying master, [15822ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [15824ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [15826ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [16002ms] querying master, [16002ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [16004ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [16006ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [18381ms] querying master, [18381ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [18383ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [18386ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [18821ms] querying master, [18822ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [18823ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [18826ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [19262ms] querying master, [19262ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [19263ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [19267ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [23082ms] querying master, [23082ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [23083ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [23086ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [26062ms] querying master, [26063ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [26064ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [26067ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [29721ms] querying master, [29722ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [29723ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [29726ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [30141ms] querying master, [30143ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > , >> > [30144ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [30147ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed, [33361ms] trace too long, truncated) >> > CAUSED BY: NoLeaderFoundException: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed >> > CAUSED BY: RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > ] >> > Connection closed >> > >> > I also added to Impala Command Line Argument Advanced >> > Configuration Snippet (Safety Valve) the setting: >> > -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx. >> com:7051 >> > >> > >> > And to Impala Service Environment Advanced Configuration >> > Snippet (Safety Valve) the setting: >> > IMPALA_KUDU=1 >> > >> > I don't think these are correct because it's not working. >> > >> > Cheers, >> > Ben >> > >> > >> > On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon > > > wrote: >> > >> > Hi Ben, >> > >> > What error are you getting? The Impala shell shouldn't be >> > relevant here -- it's just submitting queries to the >> > Impala daemons themselves. >> > >> > This deployment scenario hasn't been tested much as far as >> > I know, so I wouldn't be surprised if there are issues >> > with scheduling fragments given lack of locality, etc, but >> > I would expect it to at least "basically work". (I've done >> > it once or twice to copy data from one cluster to another) >> > >> > -Todd >> > >> > On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim >> > > wrote: >> > >> > Can someone help me with configuring Impala using >> > Cloudera Manager for Kudu 1.4.0 on CDH 5.12.0? I >> > cannot get it to connect using impala shell. >> > >> > Cheers, >> > Ben >> > >> > >> > >> > >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > >> > >> > >> > >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > >> >> -- Todd Lipcon Software Engineer, Cloudera --f403043c3a24c2e7940556e59f53 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Aug 16, 2017 at 6:16 AM, Benjamin Kim <bbuild11@gmail.com= > wrote:
Hi,

I found 2 i= ssues. First, network connection is blocked. I filed a request to open it. = Second, Kudu 1.4 had problems registering any tablet server. So, I reverted= back to 1.3.1. The error had to do security and how IP routing was not tru= sted.

Can you clarify what problem you= are hitting? Is this the security improvement that prevents from running w= ith public IP addresses without security enabled? You can override this if = you're sure that your network is secured by using the --trusted-subnets= option (see the security docs for details)

-Todd<= /div>
=C2=A0

Also, I installed = Kudu onto another cluster where I knew the network would not be a problem, = and it worked.

Cheers,
Ben

On Wed, Aug 16, 2017 at 12:44 AM Alexey= Serbin <aserb= in@cloudera.com> wrote:
Ben,

As Todd mentioned, it might be some network connectivity problem. I
would suspect some issues with connectivity between the node where the
Impala shell is running and the Kudu master node.

To start troubleshooting, I would verify that the node where you run the Impala shell (that's 172.35.120.191, right?) can establish a TCP
connection to the master RPC end-point.=C2=A0 E.g., try to run from the
command shell at 172.35.120.191:

=C2=A0 =C2=A0telnet 172.35.121.101 7051

Would it succeed?

Also, if running multi-master Kudu cluster, it might happen that masters cannot communicate with each other.=C2=A0 To troubleshoot that, I would try=
to establish a TCP connection to the RPC end-point of the master at one
node from another master node.=C2=A0 E.g., if using telnet, from
<master_node_0>, in the command-line shell:

=C2=A0 =C2=A0telnet <master_node_1> 7051
=C2=A0 =C2=A0(just substitute <master_node_0> and <master_node_1&g= t; with appropriate
hostnames/IP addresses).



Best regards,

Alexey


On 8/15/17 9:53 PM, Benjamin Kim wrote:
> Todd,
>
> Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: M= aster
> config (prod-dc1-datanode151.pdc1i.gra= dientx.com:7051
> <http://prod-dc1-datanode151.pdc1i= .gradientx.com:7051/>) has no
> leader. Exceptions received:
> org.apache.kudu.client.RecoverableException: [Peer Kudu Master -<= br> > prod-dc1-datanode151.pdc1i.gradientx= .com:7051
> <http://prod-dc1-datanode151.pdc1i= .gradientx.com:7051/>] Connection
> reset on [id: 0x6232f33f, /172.35.120.191:47848
> <http://172.35.120.191:47848/> :> /172.35.121.101:705= 1
> <http://172.35.121.101:7051/>]
>
> We got this error trying to use Kudu from within the cluster. Do you > know what this means?
>
> Cheers,
> Ben
>
>
> On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <todd@cloudera.com
> <mailto:todd= @cloudera.com>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Is there a possibility that the remote node (prod-d= c1-datanode151)
>=C2=A0 =C2=A0 =C2=A0is firewalled off from whatever host you are submit= ting the query
>=C2=A0 =C2=A0 =C2=A0to? The error message is admittedly pretty bad, but= it basically
>=C2=A0 =C2=A0 =C2=A0means it's getting "connection refused&quo= t;, indicating that either
>=C2=A0 =C2=A0 =C2=A0there is no master running on that host or it has b= een blocked (eg
>=C2=A0 =C2=A0 =C2=A0an iptables REJECT rule)
>
>=C2=A0 =C2=A0 =C2=A0-Todd
>
>=C2=A0 =C2=A0 =C2=A0On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <= bbuild11@gmail.com<= /a>
>=C2=A0 =C2=A0 =C2=A0<mailto:bbuild11@gmail.com>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Hi Todd,
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I tried to create a Kudu table using = impala shell, and I got
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0this error.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0create table my_first_table
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0id bigint,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0name string,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0primary key(id)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0partition by hash partitions 16
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0stored as kudu;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Query: create table my_first_table >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0id bigint,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0name string,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0primary key(id)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0partition by hash partitions 16
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0stored as kudu
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ERROR: ImpalaRuntimeException: Error = creating Kudu table
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'impala::default.my_first_ta= ble'
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CAUSED BY: NonRecoverableException: T= oo many attempts:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0KuduRpc(method=3DListTables, tablet= =3Dnull, attempt=3D101,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0DeadlineTracker(timeout=3D180000= , elapsed=3D178226), Traces: [0ms]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0querying master, [1ms] Sub rpc: Conne= ctToMaster sending RPC to
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[2ms] Sub rpc: ConnectToMaster receiv= ed from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [5ms] delaying RPC= due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [21ms] querying ma= ster, [22ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[24ms] Sub rpc: ConnectToMaster recei= ved from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [26ms] delaying RP= C due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [41ms] querying ma= ster, [41ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[43ms] Sub rpc: ConnectToMaster recei= ved from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [46ms] delaying RP= C due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [62ms] querying ma= ster, [62ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[64ms] Sub rpc: ConnectToMaster recei= ved from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [66ms] delaying RP= C due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [81ms] querying ma= ster, [81ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[84ms] Sub rpc: ConnectToMaster recei= ved from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [86ms] delaying RP= C due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [122ms] querying m= aster, [122ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[123ms] Sub rpc: ConnectToMaster rece= ived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [124ms] delaying R= PC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [161ms] querying m= aster, [162ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[163ms] Sub rpc: ConnectToMaster rece= ived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [166ms] delaying R= PC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [302ms] querying m= aster, [302ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[304ms] Sub rpc: ConnectToMaster rece= ived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [306ms] delaying R= PC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [382ms] querying m= aster, [382ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[384ms] Sub rpc: ConnectToMaster rece= ived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [386ms] delaying R= PC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [622ms] querying m= aster, [622ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[623ms] Sub rpc: ConnectToMaster rece= ived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [626ms] delaying R= PC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [701ms] querying m= aster, [702ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[703ms] Sub rpc: ConnectToMaster rece= ived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [706ms] delaying R= PC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [2062ms] querying = master, [2062ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[2065ms] Sub rpc: ConnectToMaster rec= eived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [2068ms] delaying = RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [4661ms] querying = master, [4662ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[4663ms] Sub rpc: ConnectToMaster rec= eived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [4666ms] delaying = RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [6961ms] querying = master, [6962ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[6964ms] Sub rpc: ConnectToMaster rec= eived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [6967ms] delaying = RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [9102ms] querying = master, [9102ms] Sub rpc:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ConnectToMaster sending RPC to server=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[9104ms] Sub rpc: ConnectToMaster rec= eived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [9106ms] delaying = RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [12001ms] querying= master, [12002ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[12003ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [12006ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [15822ms] querying= master, [15822ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[15824ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [15826ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [16002ms] querying= master, [16002ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[16004ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [16006ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [18381ms] querying= master, [18381ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[18383ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [18386ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [18821ms] querying= master, [18822ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[18823ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [18826ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [19262ms] querying= master, [19262ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[19263ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [19267ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [23082ms] querying= master, [23082ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[23083ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [23086ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [26062ms] querying= master, [26063ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[26064ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [26067ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [29721ms] querying= master, [29722ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[29723ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [29726ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [30141ms] querying= master, [30143ms] Sub
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rpc: ConnectToMaster sending RPC to s= erver
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;,
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[30144ms] Sub rpc: ConnectToMaster re= ceived from server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0response Network error: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [30147ms] delaying= RPC due to Service
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unavailable: Master config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed, [33361ms] trace to= o long, truncated)
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CAUSED BY: NoLeaderFoundException: Ma= ster config
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(prod-dc1-= datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0leader. Exceptions received:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.kudu.client.Recoverab= leException: [Peer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CAUSED BY: RecoverableException: [Pee= r
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mas= ter-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051&= gt;]
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Connection closed
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I also added to Impala Command Line A= rgument Advanced
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Configuration Snippet (Safety Valve) = the setting:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-kudu_master_hosts=3Dprod-dc1-datanode151.pdc1i.gradientx.com:7051
= >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<http:/= /prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0And to Impala Service Environment Adv= anced Configuration
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Snippet (Safety Valve) the setting: >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0IMPALA_KUDU=3D1
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I don't think these are correct b= ecause it's not working.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Cheers,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Ben
>
>
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0On Mon, Aug 14, 2017 at 9:58 PM Todd = Lipcon <todd@clou= dera.com
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<mailto:todd@cloudera.com>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Hi Ben,
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0What error are you gett= ing? The Impala shell shouldn't be
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0relevant here -- it'= ;s just submitting queries to the
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Impala daemons themselv= es.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0This deployment scenari= o hasn't been tested much as far as
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I know, so I wouldn'= ;t be surprised if there are issues
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with scheduling fragmen= ts given lack of locality, etc, but
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I would expect it to at= least "basically work". (I've done
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0it once or twice to cop= y data from one cluster to another)
>
 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-Todd
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0On Mon, Aug 14, 2017 at= 7:59 PM, Benjamin Kim
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<bbuild11@gmail.com <mailto:bbuild11@gmail.com&= gt;> wrote:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Can someo= ne help me with configuring Impala using
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Cloudera = Manager for Kudu 1.4.0 on CDH 5.12.0? I
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cannot ge= t it to connect using impala shell.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Cheers, >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Ben
>
>
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Todd Lipcon
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Software Engineer, Clou= dera
>
>
>
>
>=C2=A0 =C2=A0 =C2=A0--
>=C2=A0 =C2=A0 =C2=A0Todd Lipcon
>=C2=A0 =C2=A0 =C2=A0Software Engineer, Cloudera
>




--
Todd Lipcon
Soft= ware Engineer, Cloudera
--f403043c3a24c2e7940556e59f53--