kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lee King <yuyunliu...@gmail.com>
Subject Re: Service unavailable: Transaction failed, tablet 2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036) has exceeded its limit (67108864) or the limit of an ancestral tracker
Date Wed, 06 Sep 2017 13:11:20 GMT
I use kudu command line tool cluster ksck, the error detail:
Fetched info from all 10 Tablet Servers
Tablet 2758e5c68e974b92a3060db8575f3621 of table 'hwx_log' is
under-replicated: 1 replica(s) not RUNNING
  067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): bad state
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: Service unavailable: Transaction failed, tablet
2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036)
has exceeded its limit (67108864) or the limit of an ancestral tracker
  ad1ea284caff4b07a705c9156b0811cd (cloud-ocean-kudu-01:7050): RUNNING
  c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING
[LEADER]

3 replicas' active configs differ from the master's.
  All the peers reported by the master and tablet servers are:
  A = 067e1e7245154f0fb2720dec6c77feec
  B = ad1ea284caff4b07a705c9156b0811cd
  C = c4ed5cb73f5644a8804d3abc976d02f8

The consensus matrix is:
 Config source |    Voters    | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B   C*   |              |              | Yes
 A             | A   B   C    | 10           | 13049        | Yes
 B             | A   B   C    | 10           | 13049        | Yes
 C             | A       C    | 10           | 13692        | No
Tablet 5f996309d53445769c7db89abaa05845 of table 'hwx_log' is
under-replicated: 1 replica(s) not RUNNING
  067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): RUNNING
[LEADER]
  43d450fd58794f008caca58d44a35ff8 (cloud-ocean-kudu-03:7050): bad state
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: Service unavailable: Transaction failed, tablet
5f996309d53445769c7db89abaa05845 transaction memory consumption (66844571)
has exceeded its limit (67108864) or the limit of an ancestral tracker
  c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING

1 replicas' active configs differ from the master's.
  All the peers reported by the master and tablet servers are:
  A = 067e1e7245154f0fb2720dec6c77feec
  B = 43d450fd58794f008caca58d44a35ff8
  C = c4ed5cb73f5644a8804d3abc976d02f8

The consensus matrix is:
 Config source |    Voters    | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A*  B   C    |              |              | Yes
 A             | A*  B   C    | 12           | 7432         | Yes
 B             | A   B   C    | 11           | 7432         | Yes
 C             | A*  B   C    | 12           | 7432         | Yes
Table hwx_log has 2 under-replicated tablet(s)

the tablet cann't be add_replica or remove_replica, and I try to delete the
 data of current partition,error also occur:
WARNINGS:
Kudu error(s) reported, first error: Timed out: Failed to write batch of
76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s):
Failed to write to server: (no server available): Write(tablet:
2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed
its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is
not leader of this config. Role: FOLLOWER. Consensus state: current_term:
10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local:
false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd"
member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050
} } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type:
VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
pending_config { opid_index: 13692 OBSOLETE_local: false peers {
permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }



Kudu error(s) reported, first error: Timed out: Failed to write batch of
76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s):
Failed to write to server: (no server available): Write(tablet:
2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed
its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is
not leader of this config. Role: FOLLOWER. Consensus state: current_term:
10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local:
false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd"
member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050
} } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type:
VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
pending_config { opid_index: 13692 OBSOLETE_local: false peers {
permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
Error in Kudu table 'hwx_log': Timed out: Failed to write batch of 76249
ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s): Failed
to write to server: (no server available): Write(tablet:
2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed
its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is
not leader of this config. Role: FOLLOWER. Consensus state: current_term:
10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local:
false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd"
member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050
} } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type:
VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
pending_config { opid_index: 13692 OBSOLETE_local: false peers {
permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } } (1 of 76249
similar)

2017-09-06 14:04 GMT+08:00 Lee King <yuyunliuhen@gmail.com>:

> We got an error about :Service unavailable: Transaction failed, tablet
> 2758e5c68e974b92a3060db8575f3621 transaction memory consumption
> (67031036) has exceeded its limit (67108864) or the limit of an ancestral
> tracker.It looks like https://issues.apache.org/jira/browse/KUDU-1912.
> and the bug will be fix at 1.5,but out version is 1.4,Is there any affect
> for kudu stablity or data consistency?
>

Mime
View raw message