kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Kim <bbuil...@gmail.com>
Subject Re: Configure Impala for Kudu on Separate Cluster
Date Wed, 16 Aug 2017 13:16:16 GMT
Hi,

I found 2 issues. First, network connection is blocked. I filed a request
to open it. Second, Kudu 1.4 had problems registering any tablet server.
So, I reverted back to 1.3.1. The error had to do security and how IP
routing was not trusted.

Also, I installed Kudu onto another cluster where I knew the network would
not be a problem, and it worked.

Cheers,
Ben

On Wed, Aug 16, 2017 at 12:44 AM Alexey Serbin <aserbin@cloudera.com> wrote:

> Ben,
>
> As Todd mentioned, it might be some network connectivity problem. I
> would suspect some issues with connectivity between the node where the
> Impala shell is running and the Kudu master node.
>
> To start troubleshooting, I would verify that the node where you run the
> Impala shell (that's 172.35.120.191, right?) can establish a TCP
> connection to the master RPC end-point.  E.g., try to run from the
> command shell at 172.35.120.191:
>
>    telnet 172.35.121.101 7051
>
> Would it succeed?
>
> Also, if running multi-master Kudu cluster, it might happen that masters
> cannot communicate with each other.  To troubleshoot that, I would try
> to establish a TCP connection to the RPC end-point of the master at one
> node from another master node.  E.g., if using telnet, from
> <master_node_0>, in the command-line shell:
>
>    telnet <master_node_1> 7051
>    (just substitute <master_node_0> and <master_node_1> with appropriate
> hostnames/IP addresses).
>
>
>
> Best regards,
>
> Alexey
>
>
> On 8/15/17 9:53 PM, Benjamin Kim wrote:
> > Todd,
> >
> > Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master
> > config (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>) has no
> > leader. Exceptions received:
> > org.apache.kudu.client.RecoverableException: [Peer Kudu Master -
> > prod-dc1-datanode151.pdc1i.gradientx.com:7051
> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>] Connection
> > reset on [id: 0x6232f33f, /172.35.120.191:47848
> > <http://172.35.120.191:47848/> :> /172.35.121.101:7051
> > <http://172.35.121.101:7051/>]
> >
> > We got this error trying to use Kudu from within the cluster. Do you
> > know what this means?
> >
> > Cheers,
> > Ben
> >
> >
> > On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <todd@cloudera.com
> > <mailto:todd@cloudera.com>> wrote:
> >
> >     Is there a possibility that the remote node (prod-dc1-datanode151)
> >     is firewalled off from whatever host you are submitting the query
> >     to? The error message is admittedly pretty bad, but it basically
> >     means it's getting "connection refused", indicating that either
> >     there is no master running on that host or it has been blocked (eg
> >     an iptables REJECT rule)
> >
> >     -Todd
> >
> >     On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuild11@gmail.com
> >     <mailto:bbuild11@gmail.com>> wrote:
> >
> >         Hi Todd,
> >
> >         I tried to create a Kudu table using impala shell, and I got
> >         this error.
> >
> >         create table my_first_table
> >         (
> >           id bigint,
> >           name string,
> >           primary key(id)
> >         )
> >         partition by hash partitions 16
> >         stored as kudu;
> >         Query: create table my_first_table
> >         (
> >           id bigint,
> >           name string,
> >           primary key(id)
> >         )
> >         partition by hash partitions 16
> >         stored as kudu
> >         ERROR: ImpalaRuntimeException: Error creating Kudu table
> >         'impala::default.my_first_table'
> >         CAUSED BY: NonRecoverableException: Too many attempts:
> >         KuduRpc(method=ListTables, tablet=null, attempt=101,
> >         DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms]
> >         querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to
> >         server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [2ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [5ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [21ms] querying master, [22ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [24ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [26ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [41ms] querying master, [41ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [43ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [46ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [62ms] querying master, [62ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [64ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [66ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [81ms] querying master, [81ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [84ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [86ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [122ms] querying master, [122ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [123ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [124ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [161ms] querying master, [162ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [163ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [166ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [302ms] querying master, [302ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [304ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [306ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [382ms] querying master, [382ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [384ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [386ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [622ms] querying master, [622ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [623ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [626ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [701ms] querying master, [702ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [703ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [706ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [2062ms] querying master, [2062ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [2065ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [2068ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [4661ms] querying master, [4662ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [4663ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [4666ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [6961ms] querying master, [6962ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [6964ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [6967ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [9102ms] querying master, [9102ms] Sub rpc:
> >         ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [9104ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [9106ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [12001ms] querying master, [12002ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [12003ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [12006ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [15822ms] querying master, [15822ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [15824ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [15826ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [16002ms] querying master, [16002ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [16004ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [16006ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [18381ms] querying master, [18381ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [18383ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [18386ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [18821ms] querying master, [18822ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [18823ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [18826ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [19262ms] querying master, [19262ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [19263ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [19267ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [23082ms] querying master, [23082ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [23083ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [23086ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [26062ms] querying master, [26063ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [26064ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [26067ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [29721ms] querying master, [29722ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [29723ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [29726ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [30141ms] querying master, [30143ms] Sub
> >         rpc: ConnectToMaster sending RPC to server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
> >         [30144ms] Sub rpc: ConnectToMaster received from server
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >         response Network error: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [30147ms] delaying RPC due to Service
> >         unavailable: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed, [33361ms] trace too long, truncated)
> >         CAUSED BY: NoLeaderFoundException: Master config
> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
> >         leader. Exceptions received:
> >         org.apache.kudu.client.RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed
> >         CAUSED BY: RecoverableException: [Peer
> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
> >         Connection closed
> >
> >         I also added to Impala Command Line Argument Advanced
> >         Configuration Snippet (Safety Valve) the setting:
> >         -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx.com:7051
> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>
> >
> >         And to Impala Service Environment Advanced Configuration
> >         Snippet (Safety Valve) the setting:
> >         IMPALA_KUDU=1
> >
> >         I don't think these are correct because it's not working.
> >
> >         Cheers,
> >         Ben
> >
> >
> >         On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <todd@cloudera.com
> >         <mailto:todd@cloudera.com>> wrote:
> >
> >             Hi Ben,
> >
> >             What error are you getting? The Impala shell shouldn't be
> >             relevant here -- it's just submitting queries to the
> >             Impala daemons themselves.
> >
> >             This deployment scenario hasn't been tested much as far as
> >             I know, so I wouldn't be surprised if there are issues
> >             with scheduling fragments given lack of locality, etc, but
> >             I would expect it to at least "basically work". (I've done
> >             it once or twice to copy data from one cluster to another)
> >
> >             -Todd
> >
> >             On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim
> >             <bbuild11@gmail.com <mailto:bbuild11@gmail.com>> wrote:
> >
> >                 Can someone help me with configuring Impala using
> >                 Cloudera Manager for Kudu 1.4.0 on CDH 5.12.0? I
> >                 cannot get it to connect using impala shell.
> >
> >                 Cheers,
> >                 Ben
> >
> >
> >
> >
> >             --
> >             Todd Lipcon
> >             Software Engineer, Cloudera
> >
> >
> >
> >
> >     --
> >     Todd Lipcon
> >     Software Engineer, Cloudera
> >
>
>

Mime
View raw message