impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Zeyliger (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-5926: Avoid printing expensive stack when closing a session
Date Thu, 14 Sep 2017 19:15:08 GMT
Philip Zeyliger has posted comments on this change.

Change subject: IMPALA-5926: Avoid printing expensive stack when closing a session
......................................................................


Patch Set 3:

I took the liberty of trawling through some large impalad log files to see where else this
sort of thing could be happening. What you're seeing below is the stack traces produced by
status.cc from about 1GB of log files of impala on a cluster at work. I've removed the timestamps,
removed the hostnames (outside of the script below), and the query ids.

The punch line is that "Not implemented for this format" sure seems to come up a lot and may
benefit from similar treatment.

I'm not proud of it, but here's the "query":

find . -type f -name '*IMPALAD*.gz' | xargs gzcat  | awk '/^I/ { if(x) { print x; } x = ""
} /status.cc/ { x=" "; } { if(x) { x=x  $0 } }'  | sed -e 's/0x[0-9a-fx]* //g' | sed -e 's/[0-9a-f]\{16\}:[0-9a-f]*/QUERYID/g'
|  tr -s '\t' ' ' | tr '[0-9]' 'N' | sort | uniq -c  | sort -n | tee output.txt


300090  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Not implemented for this format. @ impala::Status::Status()
@ impala::HdfsScanNodeBase::Codegen() @ impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen()
@ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
70111  INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Not implemented for this format. @ impala::Status::Status()
@ impala::HdfsScanNodeBase::Codegen() @ impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen()
@ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
52119  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Session closed @ impala::Status::Status()
@ impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::ConnectionEnd() @ impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
50797  INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Not implemented for this format. @ impala::Status::Status()
@ impala::HdfsScanNodeBase::Codegen() @ impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen()
@ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
43115  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Not implemented for this format. @ impala::Status::Status()
@ impala::HdfsScanNodeBase::Codegen() @ impala::ExecNode::Codegen() @ impala::TopNNode::Codegen()
@ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
21064  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Session closed @ impala::Status::Status()
@ impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::CloseSession() @ apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @ apache::thrift::TDispatchProcessor::process()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
16750  INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Session closed @ impala::Status::Status()
@ impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::ConnectionEnd() @ impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
10784  INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Not implemented for this format. @ impala::Status::Status()
@ impala::HdfsScanNodeBase::Codegen() @ impala::ExecNode::Codegen() @ impala::TopNNode::Codegen()
@ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
9112  INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Session closed @ impala::Status::Status()
@ impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::ConnectionEnd() @ impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
7855  INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Not implemented for this format. @ impala::Status::Status()
@ impala::HdfsScanNodeBase::Codegen() @ impala::ExecNode::Codegen() @ impala::TopNNode::Codegen()
@ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
5700  INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Session closed @ impala::Status::Status()
@ impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::CloseSession() @ apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @ apache::thrift::TDispatchProcessor::process()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
4761  INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Session closed @ impala::Status::Status()
@ impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::CloseSession() @ apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @ apache::thrift::TDispatchProcessor::process()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
4080  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNN RPC recv timed out: Client hostname:NNNNN
timed-out during recv call. @ impala::Status::Status() @ impala::DataStreamSender::Channel::DoTransmitDataRpc()
@ impala::DataStreamSender::Channel::TransmitDataHelper() @ impala::DataStreamSender::Channel::TransmitData()
@ impala::ThreadPool<>::WorkerThread() @ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run()
@ (unknown) @ start_thread @ clone
3976  INNNN NN:NN:NN.NNNNNN NNN status.cc:NNNN Not implemented for this format. @ impala::Status::Status()
@ impala::HdfsScanNodeBase::Codegen() @ impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen()
@ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
3421  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Timestamp not yet supported for codegen.
@ impala::Status::Status() @ impala::HdfsScanner::CodegenWriteCompleteTuple() @ impala::HdfsTextScanner::Codegen()
@ impala::HdfsScanNodeBase::Codegen() @ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec()
@ impala::QueryState::ExecFInstance() @ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run()
@ (unknown) @ start_thread @ clone
3227  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNN ReportExecStatus(): Received report for unknown
query ID (probably closed or cancelled): QUERYID @ impala::Status::Status() @ impala::ImpalaServer::ReportExecStatus()
@ impala::ImpalaInternalServiceProcessor::process_ReportExecStatus() @ impala::ImpalaInternalServiceProcessor::dispatchCall()
@ apache::thrift::TDispatchProcessor::process() @ apache::thrift::server::TAcceptQueueServer::Task::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
2335  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Invalid session id: QUERYID @ impala::Status::Status()
@ impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::CloseSession() @ apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @ apache::thrift::TDispatchProcessor::process()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
2016  INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Timestamp not yet supported for codegen.
@ impala::Status::Status() @ impala::HdfsScanner::CodegenWriteCompleteTuple() @ impala::HdfsTextScanner::Codegen()
@ impala::HdfsScanNodeBase::Codegen() @ impala::FragmentInstanceState::Prepare() @ impala::FragmentInstanceState::Exec()
@ impala::QueryState::ExecFInstance() @ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run()
@ (unknown) @ start_thread @ clone
1404  INNNN NN:NN:NN.NNNNNN NNN status.cc:NNNN Session closed @ impala::Status::Status() @
impala::ImpalaServer::CloseSessionInternal() @ impala::ImpalaServer::ConnectionEnd() @ impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
@ apache::thrift::server::TThreadPoolServer::Task::run() @ apache::thrift::concurrency::ThreadManager::Worker::run()
@ impala::ThriftThread::RunRunnable() @ boost::detail::function::void_function_obj_invokerN<>::invoke()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
1384  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN couldn't deserialize thrift msg:No more data
to read. @ impala::Status::Status() @ impala::DeserializeThriftMsg<>() @ impala::BaseScalarColumnReader::ReadPageHeader()
@ impala::BaseScalarColumnReader::ReadDataPage() @ impala::BaseScalarColumnReader::NextPage()
@ impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch() @ impala::HdfsParquetScanner::AssembleRows()
@ impala::HdfsParquetScanner::GetNextInternal() @ impala::HdfsParquetScanner::ProcessSplit()
@ impala::HdfsScanNode::ProcessSplit() @ impala::HdfsScanNode::ScannerThread() @ impala::Thread::SuperviseThread()
@ boost::detail::thread_data<>::run() @ (unknown) @ start_thread @ clone
1358  INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNN RPC recv timed out: Client hostname:NNNNN
timed-out during recv call. @ impala::Status::Status() @ impala::ClientConnection<>::DoRpc<>()
@ impala::DataStreamSender::Channel::DoTransmitDataRpc() @ impala::DataStreamSender::Channel::TransmitDataHelper()
@ impala::DataStreamSender::Channel::TransmitData() @ impala::ThreadPool<>::WorkerThread()
@ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @ (unknown)
@ start_thread @ clone
1031  INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNN RPC recv timed out: Client hostname:NNNNN
timed-out during recv call. @ impala::Status::Status() @ impala::DataStreamSender::Channel::DoTransmitDataRpc()
@ impala::DataStreamSender::Channel::TransmitDataHelper() @ impala::DataStreamSender::Channel::TransmitData()
@ impala::ThreadPool<>::WorkerThread() @ impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run()
@ (unknown) @ start_thread @ clone
1008  INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN couldn't deserialize thrift msg:No more
data to read. @ impala::Status::Status() @ impala::DeserializeThriftMsg<>() @ impala::BaseScalarColumnReader::ReadPageHeader()
@ impala::BaseScalarColumnReader::ReadDataPage() @ impala::BaseScalarColumnReader::NextPage()
@ impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch() @ impala::HdfsParquetScanner::AssembleRows()
@ impala::HdfsParquetScanner::GetNextInternal() @ impala::HdfsParquetScanner::ProcessSplit()
@ impala::HdfsScanNode::ProcessSplit() @ impala::HdfsScanNode::ScannerThread() @ impala::Thread::SuperviseThread()
@ boost::detail::thread_data<>::run() @ (unknown) @ start_thread @ clone

-- 
To view, visit http://gerrit.cloudera.org:8080/8060
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I9d0ca485dd17bca758d916040745288c1a20c69f
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Mostafa Mokhtar <mmokhtar@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <philip@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-HasComments: No

Mime
View raw message