kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Kim <bbuil...@gmail.com>
Subject Re: Configure Impala for Kudu on Separate Cluster
Date Wed, 16 Aug 2017 04:53:08 GMT
Todd,

Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master
config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
Kudu Master - prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
reset on [id: 0x6232f33f, /172.35.120.191:47848 :> /172.35.121.101:7051]

We got this error trying to use Kudu from within the cluster. Do you know
what this means?

Cheers,
Ben


On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <todd@cloudera.com> wrote:

> Is there a possibility that the remote node (prod-dc1-datanode151) is
> firewalled off from whatever host you are submitting the query to? The
> error message is admittedly pretty bad, but it basically means it's getting
> "connection refused", indicating that either there is no master running on
> that host or it has been blocked (eg an iptables REJECT rule)
>
> -Todd
>
> On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuild11@gmail.com> wrote:
>
>> Hi Todd,
>>
>> I tried to create a Kudu table using impala shell, and I got this error.
>>
>> create table my_first_table
>> (
>>   id bigint,
>>   name string,
>>   primary key(id)
>> )
>> partition by hash partitions 16
>> stored as kudu;
>> Query: create table my_first_table
>> (
>>   id bigint,
>>   name string,
>>   primary key(id)
>> )
>> partition by hash partitions 16
>> stored as kudu
>> ERROR: ImpalaRuntimeException: Error creating Kudu table
>> 'impala::default.my_first_table'
>> CAUSED BY: NonRecoverableException: Too many attempts:
>> KuduRpc(method=ListTables, tablet=null, attempt=101,
>> DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms] querying
>> master, [1ms] Sub rpc: ConnectToMaster sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [5ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [21ms] querying master, [22ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [24ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [26ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [41ms] querying master, [41ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [43ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [46ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [62ms] querying master, [62ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [64ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [66ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [81ms] querying master, [81ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [84ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [86ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [122ms] querying master, [122ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [123ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [124ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [161ms] querying master, [162ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [163ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [166ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [302ms] querying master, [302ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [304ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [306ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [382ms] querying master, [382ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [384ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [386ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [622ms] querying master, [622ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [623ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [626ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [701ms] querying master, [702ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [703ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [706ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [2062ms] querying master, [2062ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2065ms]
>> Sub rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [2068ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [4661ms] querying master, [4662ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [4663ms]
>> Sub rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [4666ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [6961ms] querying master, [6962ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [6964ms]
>> Sub rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [6967ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [9102ms] querying master, [9102ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [9104ms]
>> Sub rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [9106ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [12001ms] querying master, [12002ms] Sub rpc: ConnectToMaster sending RPC
>> to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051,
>> [12003ms] Sub rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [12006ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [15822ms] querying master, [15822ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [15824ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [15826ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [16002ms] querying master, [16002ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [16004ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [16006ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [18381ms] querying master, [18381ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [18383ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [18386ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [18821ms] querying master, [18822ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [18823ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [18826ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [19262ms] querying master, [19262ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [19263ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [19267ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [23082ms] querying master, [23082ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [23083ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [23086ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [26062ms] querying master, [26063ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [26064ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [26067ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [29721ms] querying master, [29722ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [29723ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [29726ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [30141ms] querying master, [30143ms] Sub rpc: ConnectToMaster
>> sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [30144ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [30147ms] delaying RPC due to Service unavailable:
>> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no
>> leader. Exceptions received: org.apache.kudu.client.RecoverableException:
>> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
>> closed, [33361ms] trace too long, truncated)
>> CAUSED BY: NoLeaderFoundException: Master config (
>> prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions
>> received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed
>> CAUSED BY: RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed
>>
>> I also added to Impala Command Line Argument Advanced Configuration
>> Snippet (Safety Valve) the setting:
>> -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx.com:7051
>>
>> And to Impala Service Environment Advanced Configuration Snippet (Safety
>> Valve) the setting:
>> IMPALA_KUDU=1
>>
>> I don't think these are correct because it's not working.
>>
>> Cheers,
>> Ben
>>
>>
>> On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <todd@cloudera.com> wrote:
>>
>>> Hi Ben,
>>>
>>> What error are you getting? The Impala shell shouldn't be relevant here
>>> -- it's just submitting queries to the Impala daemons themselves.
>>>
>>> This deployment scenario hasn't been tested much as far as I know, so I
>>> wouldn't be surprised if there are issues with scheduling fragments given
>>> lack of locality, etc, but I would expect it to at least "basically work".
>>> (I've done it once or twice to copy data from one cluster to another)
>>>
>>> -Todd
>>>
>>> On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim <bbuild11@gmail.com>
>>> wrote:
>>>
>>>> Can someone help me with configuring Impala using Cloudera Manager for
>>>> Kudu 1.4.0 on CDH 5.12.0? I cannot get it to connect using impala shell.
>>>>
>>>> Cheers,
>>>> Ben
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message