Hi Lee,

It seems this email never got a response, sorry about that. Did you figure out a solution that you can share with the group?

Thanks

-Todd

On Wed, Sep 6, 2017 at 6:11 AM, Lee King <yuyunliuhen@gmail.com> wrote:
I use kudu command line tool cluster ksck, the error detail:
Fetched info from all 10 Tablet Servers
Tablet 2758e5c68e974b92a3060db8575f3621 of table 'hwx_log' is under-replicated: 1 replica(s) not RUNNING
  067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): bad state
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: Service unavailable: Transaction failed, tablet 2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036) has exceeded its limit (67108864) or the limit of an ancestral tracker
  ad1ea284caff4b07a705c9156b0811cd (cloud-ocean-kudu-01:7050): RUNNING
  c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING [LEADER]

3 replicas' active configs differ from the master's.
  All the peers reported by the master and tablet servers are:
  A = 067e1e7245154f0fb2720dec6c77feec
  B = ad1ea284caff4b07a705c9156b0811cd
  C = c4ed5cb73f5644a8804d3abc976d02f8

The consensus matrix is:
 Config source |    Voters    | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B   C*   |              |              | Yes
 A             | A   B   C    | 10           | 13049        | Yes
 B             | A   B   C    | 10           | 13049        | Yes
 C             | A       C    | 10           | 13692        | No
Tablet 5f996309d53445769c7db89abaa05845 of table 'hwx_log' is under-replicated: 1 replica(s) not RUNNING
  067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): RUNNING [LEADER]
  43d450fd58794f008caca58d44a35ff8 (cloud-ocean-kudu-03:7050): bad state
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: Service unavailable: Transaction failed, tablet 5f996309d53445769c7db89abaa05845 transaction memory consumption (66844571) has exceeded its limit (67108864) or the limit of an ancestral tracker
  c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING

1 replicas' active configs differ from the master's.
  All the peers reported by the master and tablet servers are:
  A = 067e1e7245154f0fb2720dec6c77feec
  B = 43d450fd58794f008caca58d44a35ff8
  C = c4ed5cb73f5644a8804d3abc976d02f8

The consensus matrix is:
 Config source |    Voters    | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A*  B   C    |              |              | Yes
 A             | A*  B   C    | 12           | 7432         | Yes
 B             | A   B   C    | 11           | 7432         | Yes
 C             | A*  B   C    | 12           | 7432         | Yes
Table hwx_log has 2 under-replicated tablet(s)

the tablet cann't be add_replica or remove_replica, and I try to delete the  data of current partition,error also occur:
WARNINGS: 
Kudu error(s) reported, first error: Timed out: Failed to write batch of 76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s): Failed to write to server: (no server available): Write(tablet: 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local: false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index: 13692 OBSOLETE_local: false peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }



Kudu error(s) reported, first error: Timed out: Failed to write batch of 76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s): Failed to write to server: (no server available): Write(tablet: 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local: false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index: 13692 OBSOLETE_local: false peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
Error in Kudu table 'hwx_log': Timed out: Failed to write batch of 76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s): Failed to write to server: (no server available): Write(tablet: 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local: false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index: 13692 OBSOLETE_local: false peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } } (1 of 76249 similar)

2017-09-06 14:04 GMT+08:00 Lee King <yuyunliuhen@gmail.com>:
We got an error about :Service unavailable: Transaction failed, tablet 2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036) has exceeded its limit (67108864) or the limit of an ancestral tracker.It looks like https://issues.apache.org/jira/browse/KUDU-1912. and the bug will be fix at 1.5,but out version is 1.4,Is there any affect for kudu stablity or data consistency?




--
Todd Lipcon
Software Engineer, Cloudera