kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Kim <bbuil...@gmail.com>
Subject Re: Configure Impala for Kudu on Separate Cluster
Date Fri, 18 Aug 2017 11:46:21 GMT
Todd,

I'll keep this in mind. This information will be useful. I'll try again.

Thanks,
Ben


On Wed, Aug 16, 2017 at 4:32 PM Todd Lipcon <todd@cloudera.com> wrote:

> On Wed, Aug 16, 2017 at 6:16 AM, Benjamin Kim <bbuild11@gmail.com> wrote:
>
>> Hi,
>>
>> I found 2 issues. First, network connection is blocked. I filed a request
>> to open it. Second, Kudu 1.4 had problems registering any tablet server.
>> So, I reverted back to 1.3.1. The error had to do security and how IP
>> routing was not trusted.
>>
>
> Can you clarify what problem you are hitting? Is this the security
> improvement that prevents from running with public IP addresses without
> security enabled? You can override this if you're sure that your network is
> secured by using the --trusted-subnets option (see the security docs for
> details)
>
> -Todd
>
>
>>
>> Also, I installed Kudu onto another cluster where I knew the network
>> would not be a problem, and it worked.
>>
>> Cheers,
>> Ben
>>
>> On Wed, Aug 16, 2017 at 12:44 AM Alexey Serbin <aserbin@cloudera.com>
>> wrote:
>>
>>> Ben,
>>>
>>> As Todd mentioned, it might be some network connectivity problem. I
>>> would suspect some issues with connectivity between the node where the
>>> Impala shell is running and the Kudu master node.
>>>
>>> To start troubleshooting, I would verify that the node where you run the
>>> Impala shell (that's 172.35.120.191, right?) can establish a TCP
>>> connection to the master RPC end-point.  E.g., try to run from the
>>> command shell at 172.35.120.191:
>>>
>>>    telnet 172.35.121.101 7051
>>>
>>> Would it succeed?
>>>
>>> Also, if running multi-master Kudu cluster, it might happen that masters
>>> cannot communicate with each other.  To troubleshoot that, I would try
>>> to establish a TCP connection to the RPC end-point of the master at one
>>> node from another master node.  E.g., if using telnet, from
>>> <master_node_0>, in the command-line shell:
>>>
>>>    telnet <master_node_1> 7051
>>>    (just substitute <master_node_0> and <master_node_1> with appropriate
>>> hostnames/IP addresses).
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Alexey
>>>
>>>
>>> On 8/15/17 9:53 PM, Benjamin Kim wrote:
>>> > Todd,
>>> >
>>> > Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master
>>> > config (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>) has no
>>> > leader. Exceptions received:
>>> > org.apache.kudu.client.RecoverableException: [Peer Kudu Master -
>>> > prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>] Connection
>>> > reset on [id: 0x6232f33f, /172.35.120.191:47848
>>> > <http://172.35.120.191:47848/> :> /172.35.121.101:7051
>>> > <http://172.35.121.101:7051/>]
>>> >
>>> > We got this error trying to use Kudu from within the cluster. Do you
>>> > know what this means?
>>> >
>>> > Cheers,
>>> > Ben
>>> >
>>> >
>>> > On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <todd@cloudera.com
>>> > <mailto:todd@cloudera.com>> wrote:
>>> >
>>> >     Is there a possibility that the remote node (prod-dc1-datanode151)
>>> >     is firewalled off from whatever host you are submitting the query
>>> >     to? The error message is admittedly pretty bad, but it basically
>>> >     means it's getting "connection refused", indicating that either
>>> >     there is no master running on that host or it has been blocked (eg
>>> >     an iptables REJECT rule)
>>> >
>>> >     -Todd
>>> >
>>> >     On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuild11@gmail.com
>>> >     <mailto:bbuild11@gmail.com>> wrote:
>>> >
>>> >         Hi Todd,
>>> >
>>> >         I tried to create a Kudu table using impala shell, and I got
>>> >         this error.
>>> >
>>> >         create table my_first_table
>>> >         (
>>> >           id bigint,
>>> >           name string,
>>> >           primary key(id)
>>> >         )
>>> >         partition by hash partitions 16
>>> >         stored as kudu;
>>> >         Query: create table my_first_table
>>> >         (
>>> >           id bigint,
>>> >           name string,
>>> >           primary key(id)
>>> >         )
>>> >         partition by hash partitions 16
>>> >         stored as kudu
>>> >         ERROR: ImpalaRuntimeException: Error creating Kudu table
>>> >         'impala::default.my_first_table'
>>> >         CAUSED BY: NonRecoverableException: Too many attempts:
>>> >         KuduRpc(method=ListTables, tablet=null, attempt=101,
>>> >         DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms]
>>> >         querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to
>>> >         server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [2ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [5ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [21ms] querying master, [22ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [24ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [26ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [41ms] querying master, [41ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [43ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [46ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [62ms] querying master, [62ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [64ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [66ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [81ms] querying master, [81ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [84ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [86ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [122ms] querying master, [122ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [123ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [124ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [161ms] querying master, [162ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [163ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [166ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [302ms] querying master, [302ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [304ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [306ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [382ms] querying master, [382ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [384ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [386ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [622ms] querying master, [622ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [623ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [626ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [701ms] querying master, [702ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [703ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [706ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [2062ms] querying master, [2062ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [2065ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [2068ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [4661ms] querying master, [4662ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [4663ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [4666ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [6961ms] querying master, [6962ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [6964ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [6967ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [9102ms] querying master, [9102ms] Sub rpc:
>>> >         ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [9104ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [9106ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [12001ms] querying master, [12002ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [12003ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [12006ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [15822ms] querying master, [15822ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [15824ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [15826ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [16002ms] querying master, [16002ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [16004ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [16006ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [18381ms] querying master, [18381ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [18383ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [18386ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [18821ms] querying master, [18822ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [18823ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [18826ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [19262ms] querying master, [19262ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [19263ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [19267ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [23082ms] querying master, [23082ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [23083ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [23086ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [26062ms] querying master, [26063ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [26064ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [26067ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [29721ms] querying master, [29722ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [29723ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [29726ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [30141ms] querying master, [30143ms] Sub
>>> >         rpc: ConnectToMaster sending RPC to server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
>>> >         [30144ms] Sub rpc: ConnectToMaster received from server
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >         response Network error: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [30147ms] delaying RPC due to Service
>>> >         unavailable: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed, [33361ms] trace too long, truncated)
>>> >         CAUSED BY: NoLeaderFoundException: Master config
>>> >         (prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has
no
>>> >         leader. Exceptions received:
>>> >         org.apache.kudu.client.RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed
>>> >         CAUSED BY: RecoverableException: [Peer
>>> >         master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
>>> >         Connection closed
>>> >
>>> >         I also added to Impala Command Line Argument Advanced
>>> >         Configuration Snippet (Safety Valve) the setting:
>>> >         -kudu_master_hosts=
>>> prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>> >         <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>
>>> >
>>> >         And to Impala Service Environment Advanced Configuration
>>> >         Snippet (Safety Valve) the setting:
>>> >         IMPALA_KUDU=1
>>> >
>>> >         I don't think these are correct because it's not working.
>>> >
>>> >         Cheers,
>>> >         Ben
>>> >
>>> >
>>> >         On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <todd@cloudera.com
>>> >         <mailto:todd@cloudera.com>> wrote:
>>> >
>>> >             Hi Ben,
>>> >
>>> >             What error are you getting? The Impala shell shouldn't be
>>> >             relevant here -- it's just submitting queries to the
>>> >             Impala daemons themselves.
>>> >
>>> >             This deployment scenario hasn't been tested much as far as
>>> >             I know, so I wouldn't be surprised if there are issues
>>> >             with scheduling fragments given lack of locality, etc, but
>>> >             I would expect it to at least "basically work". (I've done
>>> >             it once or twice to copy data from one cluster to another)
>>> >
>>> >             -Todd
>>> >
>>> >             On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim
>>> >             <bbuild11@gmail.com <mailto:bbuild11@gmail.com>>
wrote:
>>> >
>>> >                 Can someone help me with configuring Impala using
>>> >                 Cloudera Manager for Kudu 1.4.0 on CDH 5.12.0? I
>>> >                 cannot get it to connect using impala shell.
>>> >
>>> >                 Cheers,
>>> >                 Ben
>>> >
>>> >
>>> >
>>> >
>>> >             --
>>> >             Todd Lipcon
>>> >             Software Engineer, Cloudera
>>> >
>>> >
>>> >
>>> >
>>> >     --
>>> >     Todd Lipcon
>>> >     Software Engineer, Cloudera
>>> >
>>>
>>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message