kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "fengbaoli@uce.cn" <fengba...@uce.cn>
Subject kudu use impala query question!
Date Thu, 11 Oct 2018 03:11:14 GMT
Hi:

   I used CDH5.14.0,kudu version is 1.6.0 (3 kudu master,7 kudu ts), Kudu ts and yarn mix
and deploy on the same machine. 8 SAS on each TS machine (1T capacity)

   When I query the kudu table with impala, the kudu tsserver appears unavailable. The tsserver
log:
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0919 16:40:20.339299 139726 consensus_queue.cc:618] T 92d642ac75fe4fa0bdb85fd879a1e725 P
162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 400 is ahead of the local log (next sequential op:
400). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last
received: 455.400, Next index: 401, Last known committed idx: 400, Time since last communication:
0.084s
E0919 16:40:20.630578 140247 consensus_queue.cc:618] T 7f65df8fb13e467493a257aae698f219 P
162f275784fa4fbfa49ad8a2639f87c4 [NON_LEADER]: Error trying to read ahead of the log while
preparing peer request: Incomplete: Op with index 7925 is ahead of the local log (next sequential
op: 7925). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM,
Last received: 1.7924, Next index: 7926, Last known committed idx: 7925, Time since last communication:
0.001s
E0925 10:45:18.919159 33765 consensus_queue.cc:618] T a3854738cf284426bea0943184c0a4d5 P 162f275784fa4fbfa49ad8a2639f87c4
[LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete:
Op with index 380 is ahead of the local log (next sequential op: 380). Destination peer: Peer:
739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 443.380, Next index:
381, Last known committed idx: 380, Time since last communication: 0.042s
E0925 10:45:18.908238 33741 consensus_queue.cc:618] T be346255d6344e6ba12f18b1084dd2f4 P 162f275784fa4fbfa49ad8a2639f87c4
[LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete:
Op with index 74419 is ahead of the local log (next sequential op: 74419). Destination peer:
Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 346.74419, Next
index: 74420, Last known committed idx: 74419, Time since last communication: 0.019s
E0925 10:45:18.918596 33759 consensus_queue.cc:618] T 98faf1901e2848a985afb974167e0582 P 162f275784fa4fbfa49ad8a2639f87c4
[LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete:
Op with index 324 is ahead of the local log (next sequential op: 324). Destination peer: Peer:
739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 375.324, Next index:
325, Last known committed idx: 324, Time since last communication: 0.043s
E0929 16:41:43.993786 191493 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1 P
162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op:
49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last
communication: 0.536s
E0929 16:43:05.334996 196311 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1 P
162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op:
49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last
communication: 81.877s
E0929 16:43:05.337278 196314 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1 P
162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op:
49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last
communication: 81.879s
E0929 16:47:05.313355  7456 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4
[LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete:
Op with index 295 is ahead of the local log (next sequential op: 295). Destination peer: Peer:
8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last received: 371.294, Next index:
296, Last known committed idx: 295, Time since last communication: 0.606s
E0929 16:47:05.822614 12011 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4
[LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete:
Op with index 295 is ahead of the local log (next sequential op: 295). Destination peer: Peer:
8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last received: 371.294, Next index:
296, Last known committed idx: 295, Time since last communication: 1.115s
E0929 16:47:06.278580 12300 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4
[LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete:
Op with index 295 is ahead of the local log (next sequential op: 295). Destination peer: Peer:
8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last received: 371.294, Next index:
296, Last known committed idx: 295, Time since last communication: 1.571s

At this point, I use the ksck tool to output as follows:
Errors:
==================
error fetching info from tablet servers: Network error: Not all Tablet Servers are reachable
table consistency check error: Corruption: 193 out of 197 table(s) are bad

FAILED
Runtime error: ksck discovered errors


But in less than five minutes, kudu automatically returns to normal. When kudu has TS anomalies,
I use spark streaming to read Kafka data and write to kudu. When kudu returns to normal, there
is a certain probability that the written data will be lost.


Is there any good advice to help me, or to help me by modifying the parameters of kudu, or
to be careful when writing code that the data spark streaming writes is not lost




优速物流有限公司
大数据中心     冯宝利
Mobil:15050552430
Email:fengbaoli@uce.cn
Mime
View raw message