impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Mulder (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IMPALA-5415) boost termination: boost::thread_resource_error: Resource temporarily unavailable
Date Thu, 01 Jun 2017 05:29:04 GMT
Matthew Mulder created IMPALA-5415:
--------------------------------------

             Summary: boost termination: boost::thread_resource_error: Resource temporarily
unavailable
                 Key: IMPALA-5415
                 URL: https://issues.apache.org/jira/browse/IMPALA-5415
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
    Affects Versions: Impala 2.9.0
            Reporter: Matthew Mulder
            Assignee: Michael Ho


While running the TPC-DS workload with 20 concurrent threads on a 140 node cluster I encountered
two crashes, both on the same host, that look like some thread limit was reached. In one crash
gdb found 32296 threads and in the other it found 31781 threads. I see the following message
in the error log:{code}terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::thread_resource_error>
>'
  what():  boost::thread_resource_error: Resource temporarily unavailable{code}Backtrace:{code}Program
terminated with signal 6, Aborted.
#0  0x0000003c2dc32625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-gssapi-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64
glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64
libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64
openssl-1.0.1e-42.el6_7.4.x86_64 oracle-j2sdk1.7-1.7.0+update67-1.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x0000003c2dc32625 in raise () from /lib64/libc.so.6
#1  0x0000003c2dc33e05 in abort () from /lib64/libc.so.6
#2  0x00007f323cb7707d in __gnu_cxx::__verbose_terminate_handler() () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6
#3  0x00007f323cb750e6 in ?? () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6
#4  0x00007f323cb75131 in std::terminate() () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6
#5  0x00007f323cb75348 in __cxa_throw () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6
#6  0x000000000081dd4d in void boost::throw_exception<boost::thread_resource_error>(boost::thread_resource_error
const&) ()
#7  0x0000000000bedda0 in boost::thread::thread<void (*)(std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, boost::function<void ()()>, impala::Promise<long>*),
std::basic_string<char, std::char_traits<char>, std::allocator<char> >,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >,
boost::function<void ()()>, impala::Promise<long>*>(void (*)(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, boost::function<void
()()>, impala::Promise<long>*), std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, boost::function<void ()()>, impala::Promise<long>*)
()
#8  0x0000000000bea1b5 in impala::Thread::StartThread(boost::function<void ()()> const&)
()
#9  0x0000000000df2a63 in impala::DataStreamSender::DataStreamSender(impala::ObjectPool*,
int, impala::RowDescriptor const&, impala::TDataStreamSink const&, std::vector<impala::TPlanFragmentDestination,
std::allocator<impala::TPlanFragmentDestination> > const&, int) ()
#10 0x0000000000c461f7 in impala::DataSink::Create(impala::ObjectPool*, impala::TPlanFragmentCtx
const&, impala::TPlanFragmentInstanceCtx const&, impala::RowDescriptor const&,
impala::DataSink**) ()
#11 0x0000000000a7cfaf in impala::FragmentInstanceState::Prepare() ()
#12 0x0000000000a7dae1 in impala::FragmentInstanceState::Exec() ()
#13 0x0000000000a69bb6 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
()
#14 0x0000000000bec7d9 in impala::Thread::SuperviseThread(std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, boost::function<void ()()>, impala::Promise<long>*)
()
#15 0x0000000000bed194 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, boost::function<void
()()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void
()()> >, boost::_bi::value<impala::Promise<long>*> > > >::run()
()
#16 0x0000000000e540ea in ?? ()
#17 0x0000003c2e007aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003c2dce893d in clone () from /lib64/libc.so.6{code}{code}Limit                
    Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             65536                65536                processes 
Max open files            262144               262144               files     
Max locked memory         unlimited            unlimited            bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       1032751              1032751              signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us{code}

/proc/sys/kernel/threads-max 
2065503

/proc/sys/vm/max_map_count
65530

/proc/sys/kernel/pid_max
196608



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message