kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Дмитрий Павлов <dm.pav...@inbox.ru>
Subject [No Subject]
Date Tue, 02 Jul 2019 08:24:52 GMT

Hi guys

I'm encountered a strange behaviour about replication of 2 tablets in my table in Kudu cluster
The table was in UNDER REPLICATED status. So i stoped all activity on cluster to make it cold.
But even in 2 hours table was in UNDER REPLICATED state so i checked rows_updated/rows_inserted
metric and found out what replication process is very slow 1-2 K rows per second. I checked
the logs of 2 tservers where tablets were located i found following errors:
W0701 12:10:07.124253    10 kernel_stack_watchdog.cc:198] Thread 141396 stuck at /tmp/apache-kudu-1.9
Kernel stack:                                                    
[<ffffffffffffffff>] 0xffffffffffffffff                              
User stack:                                                    
    @     0x7f04971a45d0  (unknown)                                  
    @           0xb4b21c  kudu::consensus::LogCache::EvictSomeUnlocked()      
    @           0xb4bec6  kudu::consensus::LogCache::EvictThroughOp()        
    @           0xb47d2f  kudu::consensus::PeerMessageQueue::ResponseFromPeer()  
    @           0xb490b1  kudu::consensus::PeerMessageQueue::LocalPeerAppendFinished()
    @           0xb4bbcc  kudu::consensus::LogCache::LogCallback()          
    @           0xb972d2  kudu::log::Log::AppendThread::HandleGroup()        
    @           0xb97c2d  kudu::log::Log::AppendThread::DoWork()            
    @          0x1e5bdff  kudu::ThreadPool::DispatchThread()              
    @          0x1e51634  kudu::Thread::SuperviseThread()                
    @     0x7f049719cdd5  start_thread                                
    @     0x7f0495473ead  __clone  
Both tservers were restarted and replication had been finished immediately
Did anybody else encounter this issue?

Thanks Dmitry Pavlov
View raw message