kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Berkeley <wdberke...@cloudera.com>
Subject Re: kudu use impala query question!
Date Thu, 18 Oct 2018 18:19:16 GMT
I think those messages are harmless and indicative of an underlying issue
of leader elections, perhaps caused by load. The full ksck output would be
helpful to understand more.

This shouldn't cause writes to be lost. More likely the data was not
written at all- are you checking for the success of the write operations in
your code?

-Will

On Wed, Oct 10, 2018 at 8:12 PM fengbaoli@uce.cn <fengbaoli@uce.cn> wrote:

> Hi:
>
>    I used CDH5.14.0,kudu version is 1.6.0 (3 kudu master,7 kudu ts), Kudu
> ts and yarn mix and deploy on the same machine.
> 8 SAS on each TS machine (1T capacity)
>
>
> When I query the kudu table with impala, the kudu tsserver appears unavailable. The tsserver
log:
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>
> E0919 16:40:20.339299 139726 consensus_queue.cc:618] T 92d642ac75fe4fa0bdb85fd879a1e725
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 400 is ahead of the local log (next sequential op:
400). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last
received: 455.400, Next index: 401, Last known committed idx: 400, Time since last communication:
0.084s
>
> E0919 16:40:20.630578 140247 consensus_queue.cc:618] T 7f65df8fb13e467493a257aae698f219
P 162f275784fa4fbfa49ad8a2639f87c4 [NON_LEADER]: Error trying to read ahead of the log while
preparing peer request: Incomplete: Op with index 7925 is ahead of the local log (next sequential
op: 7925). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM,
Last received: 1.7924, Next index: 7926, Last known committed idx: 7925, Time since last communication:
0.001s
>
> E0925 10:45:18.919159 33765 consensus_queue.cc:618] T a3854738cf284426bea0943184c0a4d5
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 380 is ahead of the local log (next sequential op:
380). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 443.380, Next index: 381, Last known committed idx: 380, Time since last communication:
0.042s
>
> E0925 10:45:18.908238 33741 consensus_queue.cc:618] T be346255d6344e6ba12f18b1084dd2f4
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 74419 is ahead of the local log (next sequential op:
74419). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 346.74419, Next index: 74420, Last known committed idx: 74419, Time since last communication:
0.019s
>
> E0925 10:45:18.918596 33759 consensus_queue.cc:618] T 98faf1901e2848a985afb974167e0582
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 324 is ahead of the local log (next sequential op:
324). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 375.324, Next index: 325, Last known committed idx: 324, Time since last communication:
0.043s
>
> E0929 16:41:43.993786 191493 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op:
49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last
communication: 0.536s
>
> E0929 16:43:05.334996 196311 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op:
49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last
communication: 81.877s
>
> E0929 16:43:05.337278 196314 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op:
49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last
received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last
communication: 81.879s
>
> E0929 16:47:05.313355  7456 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 295 is ahead of the local log (next sequential op:
295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last
received: 371.294, Next index: 296, Last known committed idx: 295, Time since last communication:
0.606s
>
> E0929 16:47:05.822614 12011 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 295 is ahead of the local log (next sequential op:
295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last
received: 371.294, Next index: 296, Last known committed idx: 295, Time since last communication:
1.115s
>
> E0929 16:47:06.278580 12300 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34
P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing
peer request: Incomplete: Op with index 295 is ahead of the local log (next sequential op:
295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last
received: 371.294, Next index: 296, Last known committed idx: 295, Time since last communication:
1.571s
>
> At this point, I use the ksck tool to output as follows:
> Errors:
> ==================
>
> error fetching info from tablet servers: Network error: Not all Tablet Servers are reachable
> table consistency check error: Corruption: 193 out of 197 table(s) are bad
>
> FAILED
> Runtime error: ksck discovered errors
>
>
>
> But in less than five minutes, kudu automatically returns to normal. When kudu has TS
anomalies, I use spark streaming to read Kafka data and write to kudu. When kudu returns to
normal, there is a certain probability that the written data will be lost.
>
>
>
> Is there any good advice to help me, or to help me by modifying the parameters of kudu,
or to be careful when writing code that the data spark streaming writes is not lost
>
>
> ------------------------------
> [image: 说明: logo1]
> 优速物流有限公司
> 大数据中心     冯宝利
> *Mobil*:15050552430
> *Email*:fengbaoli@uce.cn
>

Mime
View raw message