kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Heo <jason.heo....@gmail.com>
Subject Re: kudu-tserver died suddenly
Date Tue, 06 Jun 2017 03:11:31 GMT
Hi, Mike

I also think the patch is not related. just wanted to let someone know the
version which I use.

I checked out with this command "git fetch
http://gerrit.cloudera.org:8080/kudu refs/changes/25/6925/11 && git
checkout FETCH_HEAD"

I've included more logs.

I0605 15:04:30.276859 138639 maintenance_manager.cc:436] P
125dbd2ffb8a401bb7e4fd982995ccf8:
UndoDeltaBlockGCOp(f50aac31d00a4ea6b8716f205a1e8f40) metrics:
{"cfile_init":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4}

I0605 15:04:30.279415 138637 tablet.cc:1512] T
f50aac31d00a4ea6b8716f205a1e8f40 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Compaction: stage 1 complete, picked 9 rowsets to compact

I0605 15:04:30.279464 138637 compaction.cc:881] Selected 9 rowsets to
compact:

I0605 15:04:30.279474 138637 compaction.cc:884] RowSet(524)(current size on
disk: ~23945114 bytes)

I0605 15:04:30.279479 138637 compaction.cc:884] RowSet(525)(current size on
disk: ~24539818 bytes)

I0605 15:04:30.279482 138637 compaction.cc:884] RowSet(526)(current size on
disk: ~24827238 bytes)

I0605 15:04:30.279485 138637 compaction.cc:884] RowSet(527)(current size on
disk: ~24766186 bytes)

I0605 15:04:30.279489 138637 compaction.cc:884] RowSet(528)(current size on
disk: ~24862612 bytes)

I0605 15:04:30.279495 138637 compaction.cc:884] RowSet(529)(current size on
disk: ~24750447 bytes)

I0605 15:04:30.279498 138637 compaction.cc:884] RowSet(530)(current size on
disk: ~24760393 bytes)

I0605 15:04:30.279502 138637 compaction.cc:884] RowSet(531)(current size on
disk: ~15210250 bytes)

I0605 15:04:30.279505 138637 compaction.cc:884] RowSet(532)(current size on
disk: ~1125877 bytes)

I0605 15:04:30.279516 138637 tablet.cc:1298] T
f50aac31d00a4ea6b8716f205a1e8f40 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot:
MvccSnapshot[committed={T|T < 6130248128052830208 or (T in
{6130248128052830208})}]

I0605 15:04:30.280143 138637 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)

I0605 15:04:30.939551 138640 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)

I0605 15:04:31.252856 138637 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)

I0605 15:04:31.967231 138640 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)

I0605 15:04:32.265857 138637 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)

I0605 15:04:33.017536 138640 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)

I0605 15:04:33.309141 138637 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)

I0605 15:04:33.361656 138635 tablet.cc:1039] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush:
entering stage 1 (old memrowset already frozen for inserts)

I0605 15:04:33.361706 138635 compaction.cc:881] Selected 1 rowsets to
compact:

I0605 15:04:33.361712 138635 compaction.cc:884] memrowset(current size on
disk: ~0 bytes)

I0605 15:04:33.361716 138635 tablet.cc:1041] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Memstore in-memory size: 763 bytes

I0605 15:04:33.361727 138635 tablet.cc:1298] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush:
entering phase 1 (flushing snapshot). Phase 1 snapshot:
MvccSnapshot[committed={T|T < 6130248381986324480 or (T in
{6130248381986324480})}]

I0605 15:04:33.361981 138635 multi_column_writer.cc:89] Opened CFile
writers for 6 column(s)

I0605 15:04:33.363195 138635 tablet.cc:1383] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush:
entering phase 2 (starting to duplicate updates in new rowsets)

I0605 15:04:33.363463 138635 tablet.cc:1438] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush
Phase 2: carrying over any updates which arrived during Phase 1

I0605 15:04:33.363481 138635 tablet.cc:1440] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Phase
2 snapshot: MvccSnapshot[committed={T|T < 6130248381986324480 or (T in
{6130248381986324480})}]

I0605 15:04:33.365365 138635 tablet.cc:1482] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush
successful on 2 rows (5123 bytes)

I0605 15:04:33.365437 138635 maintenance_manager.cc:430] Time spent running
FlushMRSOp(3cca831acf744e1daee72582b8e16dc4): real 0.004s   user 0.002s
sys 0.001s

I0605 15:04:33.365506 138635 maintenance_manager.cc:436] P
125dbd2ffb8a401bb7e4fd982995ccf8:
FlushMRSOp(3cca831acf744e1daee72582b8e16dc4) metrics: {"cfile_init":1,"data
dir 0.queue_time_us":9,"data dir 0.run_cpu_time_us":60,"data dir
0.run_wall_time_us":97,"fdatasync":29,"fdatasync_us":522,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":290,"lbm_writes_lt_1ms":40,"tcmalloc_contention_cycles":198912,"thread_start_us":200,"threads_started":1,"wal-append.queue_time_us":210}

I0605 15:04:33.370317 138638 tablet.cc:1512] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Compaction: stage 1 complete, picked 3 rowsets to compact

I0605 15:04:33.370349 138638 compaction.cc:881] Selected 3 rowsets to
compact:

I0605 15:04:33.370360 138638 compaction.cc:884] RowSet(59)(current size on
disk: ~3640 bytes)

I0605 15:04:33.370368 138638 compaction.cc:884] RowSet(60)(current size on
disk: ~700 bytes)

I0605 15:04:33.370371 138638 compaction.cc:884] RowSet(150)(current size on
disk: ~1020 bytes)

I0605 15:04:33.370379 138638 tablet.cc:1298] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot:
MvccSnapshot[committed={T|T < 6130248381986324480 or (T in
{6130248381986324480})}]

I0605 15:04:33.370710 138638 multi_column_writer.cc:89] Opened CFile
writers for 6 column(s)

I0605 15:04:33.372845 138638 tablet.cc:1383] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Compaction: entering phase 2 (starting to duplicate updates in new rowsets)

I0605 15:04:33.373126 138638 tablet.cc:1438] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Compaction Phase 2: carrying over any updates which arrived during Phase 1

I0605 15:04:33.373152 138638 tablet.cc:1440] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Phase
2 snapshot: MvccSnapshot[committed={T|T < 6130248381986324480 or (T in
{6130248381986324480})}]

I0605 15:04:33.375113 138638 tablet.cc:1482] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
Compaction successful on 73 rows (8585 bytes)

I0605 15:04:33.375154 138638 maintenance_manager.cc:430] Time spent running
CompactRowSetsOp(3cca831acf744e1daee72582b8e16dc4): real 0.005s     user
0.004s     sys 0.000s

I0605 15:04:33.375205 138638 maintenance_manager.cc:436] P
125dbd2ffb8a401bb7e4fd982995ccf8:
CompactRowSetsOp(3cca831acf744e1daee72582b8e16dc4) metrics:
{"cfile_cache_hit":28,"cfile_cache_hit_bytes":3300,"cfile_cache_miss":66,"cfile_cache_miss_bytes":5524,"cfile_init":14,"data
dir 0.queue_time_us":8,"data dir 0.run_cpu_time_us":47,"data dir
0.run_wall_time_us":101,"delta_iterators_relevant":2,"fdatasync":28,"fdatasync_us":259,"lbm_read_time_us":388,"lbm_reads_lt_1ms":122,"lbm_write_time_us":347,"lbm_writes_lt_1ms":40,"spinlock_wait_cycles":4864}

E0605 15:04:33.376554 138642 tablet.cc:1219] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Rowset
selected for compaction but not available anymore: RowSet(150)

E0605 15:04:33.376605 138642 tablet.cc:1219] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Rowset
selected for compaction but not available anymore: RowSet(59)

E0605 15:04:33.376615 138642 tablet.cc:1219] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Rowset
selected for compaction but not available anymore: RowSet(60)

F0605 15:04:33.377100 138642 tablet.cc:1222] T
3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Was
unable to find all rowsets selected for compaction

I0605 15:04:33.748011 138636 maintenance_manager.cc:430] Time spent running
UndoDeltaBlockGCOp(3cca831acf744e1daee72582b8e16dc4): real 0.006s   user
0.000s     sys 0.003s

I0605 15:04:33.748826 138636 maintenance_manager.cc:436] P
125dbd2ffb8a401bb7e4fd982995ccf8:
UndoDeltaBlockGCOp(3cca831acf744e1daee72582b8e16dc4) metrics:
{"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4,"tcmalloc_contention_cycles":136192}

I0605 15:04:35.059636 140969 consensus_queue.cc:709] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8
[LEADER]: Peer df7fc4cfb2d44e329e4cff76ccb64987 log is divergent from this
leader: its last log entry 2.2 is not in this leader's log and it has not
received anything from this leader yet. Falling back to committed index 1

I0605 15:04:35.059742 140969 consensus_queue.cc:733] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8
[LEADER]: Peer responded invalid term: Peer:
df7fc4cfb2d44e329e4cff76ccb64987, Is new: false, Last received: 1.1, Next
index: 2, Last known committed idx: 1, Last exchange result: ERROR, Needs
tablet copy: false

I0605 15:04:35.064334 74955 raft_consensus.cc:2354] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
LEADER]: Stepping down as leader of term 1

I0605 15:04:35.064350 140925 raft_consensus.cc:412] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
dac8ac3bc0754eb6b40d0e1697d0050b)

I0605 15:04:35.064354 141001 raft_consensus.cc:412] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
b846822ae026439faa359d75183c4b54)

I0605 15:04:35.064427 141044 raft_consensus.cc:412] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
b846822ae026439faa359d75183c4b54)

I0605 15:04:35.064491 141851 raft_consensus.cc:412] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
b846822ae026439faa359d75183c4b54)

I0605 15:04:35.064463 140607 raft_consensus.cc:412] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
bcde875557b54a70b5724ed5ac988caa)

I0605 15:04:35.064522 139580 raft_consensus.cc:412] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
FOLLOWER]: Starting pre-election (detected failure of leader
dac8ac3bc0754eb6b40d0e1697d0050b)

I0605 15:04:35.064555 141851 raft_consensus.cc:2315] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.521s

I0605 15:04:35.064543 140646 raft_consensus.cc:412] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
b846822ae026439faa359d75183c4b54)

I0605 15:04:35.064580 141671 raft_consensus.cc:412] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
df7fc4cfb2d44e329e4cff76ccb64987)

I0605 15:04:35.065110 74955 raft_consensus.cc:549] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
LEADER]: Becoming Follower/Learner. State: Replica:
125dbd2ffb8a401bb7e4fd982995ccf8, State: 1, Role: LEADER

I0605 15:04:35.066002 138766 raft_consensus.cc:1861] T
19e5232aec6047209f91cd42786ad053 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
LEADER]: Leader pre-election vote request: Denying vote to candidate
df7fc4cfb2d44e329e4cff76ccb64987 for term 4 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.070186 140637 raft_consensus.cc:412] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election (detected failure of leader
dac8ac3bc0754eb6b40d0e1697d0050b)

I0605 15:04:35.075618 141001 raft_consensus.cc:2315] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.506s

I0605 15:04:35.077335 140925 raft_consensus.cc:2315] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.775s

I0605 15:04:35.078053 141671 raft_consensus.cc:2315] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.676s

I0605 15:04:35.078641 139580 raft_consensus.cc:2315] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.982s

I0605 15:04:35.080402 141044 raft_consensus.cc:2315] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.970s

I0605 15:04:35.081475 140637 raft_consensus.cc:2315] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.557s

I0605 15:04:35.084607 140607 raft_consensus.cc:2315] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.721s

I0605 15:04:35.086302 141851 raft_consensus.cc:436] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"0c7963f2bfe04e9c9a687879fcf81e51" member_type: VOTER last_known_addr {
host: "host25.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr {
host: "host22.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.086418 140925 raft_consensus.cc:436] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"ff831fe6da7f4388aa633dc5d3f811c6" member_type: VOTER last_known_addr {
host: "host26.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr {
host: "host28.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.087604 140925 leader_election.cc:216] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
ff831fe6da7f4388aa633dc5d3f811c6

I0605 15:04:35.087857 139580 raft_consensus.cc:436] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr {
host: "host28.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"bcde875557b54a70b5724ed5ac988caa" member_type: VOTER last_known_addr {
host: "host23.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.087862 140646 raft_consensus.cc:2315] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 2.081s

I0605 15:04:35.087882 141671 raft_consensus.cc:436] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"df7fc4cfb2d44e329e4cff76ccb64987" member_type: VOTER last_known_addr {
host: "host27.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr {
host: "host28.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.088114 141044 raft_consensus.cc:436] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"0c7963f2bfe04e9c9a687879fcf81e51" member_type: VOTER last_known_addr {
host: "host25.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr {
host: "host22.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.088122 140637 raft_consensus.cc:436] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr {
host: "host28.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr {
host: "host22.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.088127 141001 raft_consensus.cc:436] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr {
host: "host22.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"df7fc4cfb2d44e329e4cff76ccb64987" member_type: VOTER last_known_addr {
host: "host27.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.088165 140607 raft_consensus.cc:436] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"ff831fe6da7f4388aa633dc5d3f811c6" member_type: VOTER last_known_addr {
host: "host26.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"bcde875557b54a70b5724ed5ac988caa" member_type: VOTER last_known_addr {
host: "host23.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.088374 140646 raft_consensus.cc:436] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Starting pre-election with config: opid_index: -1
OBSOLETE_local: false peers { permanent_uuid:
"b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr {
host: "host22.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"df7fc4cfb2d44e329e4cff76ccb64987" member_type: VOTER last_known_addr {
host: "host27.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid:
"125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr {
host: "host29.domain.nhnsystem.com" port: 7050 } }

I0605 15:04:35.090010 140637 leader_election.cc:216] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
dac8ac3bc0754eb6b40d0e1697d0050b

I0605 15:04:35.098152 141851 leader_election.cc:216] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
0c7963f2bfe04e9c9a687879fcf81e51

I0605 15:04:35.099189 141001 leader_election.cc:216] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
b846822ae026439faa359d75183c4b54

I0605 15:04:35.099319 141044 leader_election.cc:216] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
0c7963f2bfe04e9c9a687879fcf81e51

I0605 15:04:35.106904 141001 leader_election.cc:216] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
df7fc4cfb2d44e329e4cff76ccb64987

I0605 15:04:35.107388 140637 leader_election.cc:216] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
b846822ae026439faa359d75183c4b54

I0605 15:04:35.108178 141044 leader_election.cc:216] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
b846822ae026439faa359d75183c4b54

I0605 15:04:35.109321 140646 leader_election.cc:216] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
b846822ae026439faa359d75183c4b54

I0605 15:04:35.110162 141851 leader_election.cc:216] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
b846822ae026439faa359d75183c4b54

I0605 15:04:35.111464 74955 consensus_queue.cc:184] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8
[NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index:
1, Majority replicated index: 1, Committed index: 1, Last appended: 1.1,
Last appended by leader: 1, Current term: 1, Majority size: -1, State: 1,
Mode: NON_LEADER

I0605 15:04:35.111796 141292 consensus_peers.cc:187] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer
df7fc4cfb2d44e329e4cff76ccb64987 (host27.domain.nhnsystem.com:7050): Could
not obtain request from queue for peer: df7fc4cfb2d44e329e4cff76ccb64987.
Status: Not found: Peer not tracked or queue not in leader mode.

I0605 15:04:35.111865 139580 leader_election.cc:216] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer
dac8ac3bc0754eb6b40d0e1697d0050b

I0605 15:04:35.111913 140925 leader_election.cc:216] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
dac8ac3bc0754eb6b40d0e1697d0050b

I0605 15:04:35.111917 139580 leader_election.cc:216] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer
bcde875557b54a70b5724ed5ac988caa

I0605 15:04:35.112076 141671 leader_election.cc:216] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
df7fc4cfb2d44e329e4cff76ccb64987

I0605 15:04:35.112323 140607 leader_election.cc:216] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
ff831fe6da7f4388aa633dc5d3f811c6

I0605 15:04:35.112344 140646 leader_election.cc:216] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
df7fc4cfb2d44e329e4cff76ccb64987

I0605 15:04:35.113402 141671 leader_election.cc:216] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
dac8ac3bc0754eb6b40d0e1697d0050b

I0605 15:04:35.113420 140607 leader_election.cc:216] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer
bcde875557b54a70b5724ed5ac988caa

I0605 15:04:35.114395 138647 leader_election.cc:362] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
df7fc4cfb2d44e329e4cff76ccb64987. Message: Invalid argument: T
881134030ff347d992791e1d8db93632 P df7fc4cfb2d44e329e4cff76ccb64987 [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.115078 138647 leader_election.cc:362] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
df7fc4cfb2d44e329e4cff76ccb64987. Message: Invalid argument: T
63396a9239184c3799d53d93f0ce5d49 P df7fc4cfb2d44e329e4cff76ccb64987 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.115154 138647 leader_election.cc:362] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
df7fc4cfb2d44e329e4cff76ccb64987. Message: Invalid argument: T
351835030fc54ff79a605b20ee7741e5 P df7fc4cfb2d44e329e4cff76ccb64987 [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.115432 75021 consensus_queue.cc:733] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8
[NON_LEADER]: Peer responded invalid term: Peer:
bcde875557b54a70b5724ed5ac988caa, Is new: false, Last received: 1.1, Next
index: 2, Last known committed idx: 1, Last exchange result: ERROR, Needs
tablet copy: false

I0605 15:04:35.115860 75045 consensus_queue.cc:733] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8
[LEADER]: Peer responded invalid term: Peer:
dac8ac3bc0754eb6b40d0e1697d0050b, Is new: false, Last received: 3.3, Next
index: 4, Last known committed idx: 3, Last exchange result: ERROR, Needs
tablet copy: false

I0605 15:04:35.116771 138796 raft_consensus.cc:1861] T
cdfb5dcd98734ab3bdaeac2937b1006b P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
bcde875557b54a70b5724ed5ac988caa for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.116952 138783 raft_consensus.cc:1861] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
LEADER]: Leader pre-election vote request: Denying vote to candidate
bcde875557b54a70b5724ed5ac988caa for term 4 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.117341 138646 leader_election.cc:362] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
0c7963f2bfe04e9c9a687879fcf81e51. Message: Invalid argument: T
26f4871c17624227b8157b1a01e5e2cf P 0c7963f2bfe04e9c9a687879fcf81e51 [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.117452 74955 consensus_peers.cc:389] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer
bcde875557b54a70b5724ed5ac988caa (host23.domain.nhnsystem.com:7050):
Closing peer: bcde875557b54a70b5724ed5ac988caa

I0605 15:04:35.117959 138646 leader_election.cc:362] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
0c7963f2bfe04e9c9a687879fcf81e51. Message: Invalid argument: T
48cd3f9450ce4d2a8477b4fc17e73d1c P 0c7963f2bfe04e9c9a687879fcf81e51 [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.118019 138646 leader_election.cc:362] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
b846822ae026439faa359d75183c4b54. Message: Invalid argument: T
881134030ff347d992791e1d8db93632 P b846822ae026439faa359d75183c4b54 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.118037 138646 leader_election.cc:243] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.118077 74955 consensus_peers.cc:389] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer
df7fc4cfb2d44e329e4cff76ccb64987 (host27.domain.nhnsystem.com:7050):
Closing peer: df7fc4cfb2d44e329e4cff76ccb64987

I0605 15:04:35.118149 74955 raft_consensus.cc:2359] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Advancing to term 2

I0605 15:04:35.118224 138778 raft_consensus.cc:1861] T
270218692311420c8b6596fa7647010e P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
dac8ac3bc0754eb6b40d0e1697d0050b for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.118348 75047 raft_consensus.cc:2315] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.601s

I0605 15:04:35.118381 75047 raft_consensus.cc:2061] T
881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.118518 138755 raft_consensus.cc:1861] T
96cd3b27b43c42d7b3f5cca0757486f9 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
dac8ac3bc0754eb6b40d0e1697d0050b for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.118790 138646 leader_election.cc:362] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
b846822ae026439faa359d75183c4b54. Message: Invalid argument: T
351835030fc54ff79a605b20ee7741e5 P b846822ae026439faa359d75183c4b54 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.118806 138646 leader_election.cc:243] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.119104 138756 raft_consensus.cc:1861] T
5fa2b7db1bc54795856ef0fef58d2770 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
LEADER]: Leader pre-election vote request: Denying vote to candidate
dac8ac3bc0754eb6b40d0e1697d0050b for term 4 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.119427 138646 leader_election.cc:362] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
b846822ae026439faa359d75183c4b54. Message: Invalid argument: T
48cd3f9450ce4d2a8477b4fc17e73d1c P b846822ae026439faa359d75183c4b54 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.119443 138646 leader_election.cc:243] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.119434 138766 raft_consensus.cc:1875] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
LEADER]: Leader election vote request: Denying vote to candidate
bcde875557b54a70b5724ed5ac988caa for term 4 because replica is already
servicing an update from a current leader or another vote.

I0605 15:04:35.119475 75048 raft_consensus.cc:2315] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.911s

I0605 15:04:35.119493 75048 raft_consensus.cc:2061] T
351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.119493 138648 leader_election.cc:362] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
ff831fe6da7f4388aa633dc5d3f811c6. Message: Invalid argument: T
4203d61ed3aa4008955cf2cf023c2132 P ff831fe6da7f4388aa633dc5d3f811c6 [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.119614 75046 raft_consensus.cc:2354] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
LEADER]: Stepping down as leader of term 3

I0605 15:04:35.119638 75046 raft_consensus.cc:549] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
LEADER]: Becoming Follower/Learner. State: Replica:
125dbd2ffb8a401bb7e4fd982995ccf8, State: 1, Role: LEADER

I0605 15:04:35.119695 75046 consensus_queue.cc:184] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8
[NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index:
3, Majority replicated index: 3, Committed index: 3, Last appended: 3.3,
Last appended by leader: 3, Current term: 3, Majority size: -1, State: 1,
Mode: NON_LEADER

I0605 15:04:35.119776 138648 leader_election.cc:362] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
ff831fe6da7f4388aa633dc5d3f811c6. Message: Invalid argument: T
87e5a254f3aa4070bf138a6d08ed611d P ff831fe6da7f4388aa633dc5d3f811c6 [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.119892 75049 raft_consensus.cc:2315] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.579s

I0605 15:04:35.119910 75049 raft_consensus.cc:2061] T
48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.119997 75046 consensus_peers.cc:389] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer
bcde875557b54a70b5724ed5ac988caa (host23.domain.nhnsystem.com:7050):
Closing peer: bcde875557b54a70b5724ed5ac988caa

I0605 15:04:35.120003 138646 leader_election.cc:362] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
b846822ae026439faa359d75183c4b54. Message: Invalid argument: T
d0bb69948a5d4278b22943c7d79e8d19 P b846822ae026439faa359d75183c4b54 [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.120048 138646 leader_election.cc:362] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
b846822ae026439faa359d75183c4b54. Message: Invalid argument: T
26f4871c17624227b8157b1a01e5e2cf P b846822ae026439faa359d75183c4b54 [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.120059 138646 leader_election.cc:243] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.120159 75046 consensus_peers.cc:389] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer
dac8ac3bc0754eb6b40d0e1697d0050b (host28.domain.nhnsystem.com:7050):
Closing peer: dac8ac3bc0754eb6b40d0e1697d0050b

I0605 15:04:35.120218 75046 raft_consensus.cc:2359] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
FOLLOWER]: Advancing to term 4

I0605 15:04:35.120309 75050 raft_consensus.cc:2315] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.594s

I0605 15:04:35.120324 75050 raft_consensus.cc:2061] T
26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.120527 138648 leader_election.cc:362] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T
d0bb69948a5d4278b22943c7d79e8d19 P dac8ac3bc0754eb6b40d0e1697d0050b [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.120545 138648 leader_election.cc:243] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.120784 75051 raft_consensus.cc:2315] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.516s

I0605 15:04:35.120793 75051 raft_consensus.cc:2061] T
d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.120863 138648 leader_election.cc:362] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
bcde875557b54a70b5724ed5ac988caa. Message: Invalid argument: T
4203d61ed3aa4008955cf2cf023c2132 P bcde875557b54a70b5724ed5ac988caa [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.120875 138648 leader_election.cc:243] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.120986 75052 raft_consensus.cc:2315] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.730s

I0605 15:04:35.120995 75052 raft_consensus.cc:2061] T
4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.121003 138648 leader_election.cc:362] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 4 pre-election: Vote denied by peer
dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T
d766a7aac6294db28bc5202a064c9835 P dac8ac3bc0754eb6b40d0e1697d0050b [term 3
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 4 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.121055 138648 leader_election.cc:362] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 4 pre-election: Vote denied by peer
bcde875557b54a70b5724ed5ac988caa. Message: Invalid argument: T
d766a7aac6294db28bc5202a064c9835 P bcde875557b54a70b5724ed5ac988caa [term 3
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 4 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.121064 138648 leader_election.cc:243] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.121155 138806 raft_consensus.cc:1875] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2
FOLLOWER]: Leader election vote request: Denying vote to candidate
df7fc4cfb2d44e329e4cff76ccb64987 for term 2 because replica is already
servicing an update from a current leader or another vote.

I0605 15:04:35.121177 75053 raft_consensus.cc:2315] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 2.129s

I0605 15:04:35.121183 75053 raft_consensus.cc:2061] T
d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3
FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve
majority

I0605 15:04:35.121188 138648 leader_election.cc:362] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T
63396a9239184c3799d53d93f0ce5d49 P dac8ac3bc0754eb6b40d0e1697d0050b [term 1
FOLLOWER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.121201 138648 leader_election.cc:243] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.121321 75054 raft_consensus.cc:2315] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.513s

I0605 15:04:35.121332 138648 leader_election.cc:362] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Vote denied by peer
dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T
87e5a254f3aa4070bf138a6d08ed611d P dac8ac3bc0754eb6b40d0e1697d0050b [term 1
LEADER]: Leader pre-election vote request: Denying vote to candidate
125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either
leader or believes a valid leader to be alive.

I0605 15:04:35.121336 75054 raft_consensus.cc:2061] T
63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.121361 138648 leader_election.cc:243] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8
[CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.

I0605 15:04:35.121372 138797 raft_consensus.cc:2315] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.662s

I0605 15:04:35.121544 138797 raft_consensus.cc:1900] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2
FOLLOWER]: Leader pre-election vote request: Granting yes vote for
candidate df7fc4cfb2d44e329e4cff76ccb64987 in term 2.

I0605 15:04:35.121592 75055 raft_consensus.cc:2315] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Snoozing failure detection for election timeout plus an
additional 1.566s

I0605 15:04:35.121601 75055 raft_consensus.cc:2061] T
87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1
FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve
majority

I0605 15:04:35.445930 138806 raft_consensus.cc:921] T
55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2
FOLLOWER]: Refusing update from remote peer
df7fc4cfb2d44e329e4cff76ccb64987: Log matching property violated. Preceding
OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2
index: 2. (index mismatch)

I0605 15:04:35.549202 138806 raft_consensus.cc:921] T
fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 4
FOLLOWER]: Refusing update from remote peer
bcde875557b54a70b5724ed5ac988caa: Log matching property violated. Preceding
OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4
index: 4. (index mismatch)

I0605 15:04:35.732771 138640 multi_column_writer.cc:89] Opened CFile
writers for 8 column(s)


<End of Log>


2017-06-06 11:38 GMT+09:00 Mike Percy <mpercy@apache.org>:

> It seems unrelated. What release did you cherry-pick on top of?
>
> INFO log from around that timeframe would be useful.
>
> Mike
>
> On Mon, Jun 5, 2017 at 6:38 PM, Jason Heo <jason.heo.sde@gmail.com> wrote:
>
>> Hello.
>>
>> I'm using this patch https://gerrit.cloudera.org/#/c/6925/
>>
>> One of tservers died suddenly. Here is ERROR and FATAL log.
>>
>> E0605 15:04:33.376554 138642 tablet.cc:1219] T
>> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
>> Rowset selected for compaction but not available anymore: RowSet(150)
>>
>> E0605 15:04:33.376605 138642 tablet.cc:1219] T
>> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
>> Rowset selected for compaction but not available anymore: RowSet(59)
>>
>> E0605 15:04:33.376615 138642 tablet.cc:1219] T
>> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8:
>> Rowset selected for compaction but not available anymore: RowSet(60)
>>
>> F0605 15:04:33.377100 138642 tablet.cc:1222] T
>> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Was
>> unable to find all rowsets selected for compaction
>>
>>
>> <End of Log>
>>
>>
>> Could I know what's the problem? Feel free to ask any information to
>> resolve it.
>>
>>
>> Thank,
>>
>>
>> Jason
>>
>
>

Mime
View raw message