trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From feng D <fengi...@gmail.com>
Subject Re: Too many threads
Date Thu, 24 Dec 2015 14:31:58 GMT
Reindl Harald :
        Yes, you are right.  I make a lot of assumptions. I don't know how
to reslove this problem. I just has the log as below.

      I want to know why my trafficserver always restart. And I has 5
machines. There are three trafficserver that will always restart. The
machines are same.
The machines with 16G memory. And the thread number of the  other two
trafficserver is always 18. The config of all is  also same.  The ram size
is 2G.

    I want to get the direct to find the answer not the answer.

So, I want to know there questions:
1. Was trafficserver start many threads when running?
2. Is that the traffic server has two thread called traffic server normal?
3. There are many threads called the same name such as [NT_NET 8], this is
normal?




              Thanks!!

I just has this log:

***************************************************************************************

FATAL: ats_malloc: couldn't allocate 103283 bytes
/usr/local/ats_ssd_422/bin/traffic_server - STACK TRACE:
/usr/local/ats_ssd_422/lib/libtsutil.so.4(+0x23f38)[0x2b2aa3d74f38]
/usr/local/ats_ssd_422/lib/libtsutil.so.4(ats_malloc+0x30)[0x2b2aa3d77860]
/usr/local/ats_ssd_422/bin/traffic_server(RamCacheCLFUS::get(INK_MD5*,
Ptr<IOBufferData>*, unsigned int, unsigned int)+0x3fa)[0x66560a]
/usr/local/ats_ssd_422/bin/traffic_server(CacheVC::handleRead(int,
Event*)+0x215)[0x62eaf5]
/usr/local/ats_ssd_422/bin/traffic_server(Cache::open_write(Continuation*,
INK_MD5*, HTTPInfo*, long, INK_MD5*, CacheFragType, char*,
int)+0x45b)[0x6614bb]
/usr/local/ats_ssd_422/bin/traffic_server(CacheProcessor::open_write(Continuation*,
int, URL*, bool, HTTPHdr*, HTTPInfo*, long, CacheFragType)+0x8e)[0x62c77e]
/usr/local/ats_ssd_422/bin/traffic_server(HttpCacheSM::open_write(URL*,
HTTPHdr*, HTTPInfo*, long, bool, bool)+0x9f)[0x507f7f]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::do_cache_prepare_action(HttpCacheSM*,
HTTPInfo*, bool, bool)+0xf7)[0x518087]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x806)[0x529ab6]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x21a)[0x51cb7a]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0xce9)[0x529f99]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x2c0)[0x524ae0]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x1bb)[0x52946b]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x861)[0x529b11]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x2c0)[0x524ae0]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callback(int,
void*)+0x82)[0x5291b2]
/usr/local/ats_ssd_422/bin/traffic_server(TSHttpTxnReenable+0x244)[0x4b61e4]
/usr/local/ats_ssd_422/libexec/trafficserver/batch-purge.so(+0x1da6)[0x2b2ab4303da6]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x102)[0x524922]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x1bb)[0x52946b]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x2c0)[0x524ae0]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x1bb)[0x52946b]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x156)[0x5261c6]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::main_handler(int,
void*)+0xbd)[0x52572d]
/usr/local/ats_ssd_422/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x1c2)[0x507c22]
/usr/local/ats_ssd_422/bin/traffic_server(CacheVC::callcont(int)+0x5b)[0x5e918b]
/usr/local/ats_ssd_422/bin/traffic_server(CacheVC::openReadStartHead(int,
Event*)+0x1354)[0x656394]
/usr/local/ats_ssd_422/bin/traffic_server(CacheVC::handleReadDone(int,
Event*)+0x21f)[0x63010f]
/usr/local/ats_ssd_422/bin/traffic_server(Cache::open_read(Continuation*,
INK_MD5*, HTTPHdr*, CacheLookupHttpConfig*, CacheFragType, char*,
int)+0x671)[0x651f01]
/usr/local/ats_ssd_422/bin/traffic_server(CacheProcessor::open_read(Continuation*,
URL*, bool, HTTPHdr*, CacheLookupHttpConfig*, long,
CacheFragType)+0x87)[0x62c607]
/usr/local/ats_ssd_422/bin/traffic_server(HttpCacheSM::open_read(URL*,
HTTPHdr*, CacheLookupHttpConfig*, long)+0x81)[0x507e71]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x517d6b]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x742)[0x5299f2]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x2c0)[0x524ae0]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x1bb)[0x52946b]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x1aa)[0x52945a]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x2c0)[0x524ae0]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x1bb)[0x52946b]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x2c0)[0x524ae0]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::set_next_state()+0x1bb)[0x52946b]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x38f)[0x51d93f]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::main_handler(int,
void*)+0xbd)[0x52572d]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_api_callout(int,
void*)+0x2c0)[0x524ae0]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::state_add_to_list(int,
void*)+0x190)[0x525b30]
/usr/local/ats_ssd_422/bin/traffic_server(HttpSM::attach_client_session(HttpClientSession*,
IOBufferReader*)+0x35a)[0x525f9a]
/usr/local/ats_ssd_422/bin/traffic_server(HttpClientSession::state_keep_alive(int,
void*)+0xc9)[0x5095f9]
/usr/local/ats_ssd_422/bin/traffic_server[0x6898cb]
/usr/local/ats_ssd_422/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x1f2)[0x67ca02]
/usr/local/ats_ssd_422/bin/traffic_server(EThread::execute()+0xad3)[0x6ac523]
/usr/local/ats_ssd_422/bin/traffic_server[0x6aaf4a]
/lib64/libpthread.so.0(+0x79d1)[0x2b2aa57a59d1]
/lib64/libc.so.6(clone+0x6d)[0x2b2aa6799b6d]
[TrafficServer] using root directory '/usr/local/ats_ssd_422'
/usr/local/ats_ssd_422/bin/traffic_server - STACK TRACE:
/usr/local/ats_ssd_422/bin/traffic_server(RamCacheCLFUS::put(INK_MD5*,
IOBufferData*, unsigned int, bool, unsigned int, unsigned
int)+0xab1)[0x666791]
/usr/local/ats_ssd_422/bin/traffic_server(CacheVC::handleReadDone(int,
Event*)+0xb1e)[0x630a0e]
/usr/local/ats_ssd_422/bin/traffic_server(AIOCallbackInternal::io_complete(int,
void*)+0x195)[0x5ec935]
/usr/local/ats_ssd_422/bin/traffic_server(DiskHandler::mainAIOEvent(int,
Event*)+0x1f8)[0x672bc8]
/usr/local/ats_ssd_422/bin/traffic_server(EThread::execute()+0xad3)[0x6ac523]
/usr/local/ats_ssd_422/bin/traffic_server[0x6aaf4a]
/lib64/libpthread.so.0(+0x79d1)[0x2ba4b40bb9d1]
/lib64/libc.so.6(clone+0x6d)[0x2ba4b50afb6d]
FATAL: ats_malloc: couldn't allocate 156640 bytes
/usr/local/ats_ssd_422/bin/traffic_server - STACK TRACE:
/usr/local/ats_ssd_422/lib/libtsutil.so.4(+0x23f38)[0x2ba4b268af38]
/usr/local/ats_ssd_422/lib/libtsutil.so.4(ats_malloc+0x30)[0x2ba4b268d860]
/usr/local/ats_ssd_422/bin/traffic_server(RamCacheCLFUS::put(INK_MD5*,
IOBufferData*, unsigned int, bool, unsigned int, unsigned
int)+0xab1)[0x666791]
/usr/local/ats_ssd_422/bin/traffic_server(CacheVC::handleReadDone(int,
Event*)+0xb1e)[0x630a0e]
/usr/local/ats_ssd_422/bin/traffic_server(AIOCallbackInternal::io_complete(int,
void*)+0x195)[0x5ec935]
/usr/local/ats_ssd_422/bin/traffic_server(DiskHandler::mainAIOEvent(int,
Event*)+0x1f8)[0x672bc8]
/usr/local/ats_ssd_422/bin/traffic_server(EThread::execute()+0xad3)[0x6ac523]
/usr/local/ats_ssd_422/bin/traffic_server[0x6aaf4a]
/lib64/libpthread.so.0(+0x79d1)[0x2ba4b40bb9d1]
/lib64/libc.so.6(clone+0x6d)[0x2ba4b50afb6d]
[E. Mgmt] log ==> [TrafficManager] using root directory
'/usr/local/ats_ssd_422'
[TrafficServer] using root directory '/usr/local/ats_ssd_422'


2015-12-24 22:00 GMT+08:00 Reindl Harald <h.reindl@thelounge.net>:

> main problem: you are making a lot of assumptions while not provide any
> logs or tell something relevant about your configuration and hardware
>
> Am 24.12.2015 um 14:53 schrieb feng D:
>
>> I have  a 7-core cpu.  But there are  80 threads, so it is too many. The
>> same machine has 18 threads.
>>
>> And the  number of threads awalys change  from 55 to 80.  So  i think
>> this is abnormal.  Then traffic server will exit when  cann't get memery.
>>
>
> what is there abnormal?
>
> each parallel connection is a thread
>
> frankly on a perforking apache each connection is even a process and our
> backend mainserver goes up to 800 forker processes with 14 GB RAM
>
> I think every thread used much memery so the memery was exhausted.
>>
>
> how much memory has the machine?
> what are the logs saying (ATS as well as OS)
>
> our ATS proxy (dedicated VM) has 4096 MB RAM
> CONFIG proxy.config.cache.ram_cache.size INT 2816M
>
> 2015-12-24 18:12 GMT+08:00 Reindl Harald <h.reindl@thelounge.net
>> <mailto:h.reindl@thelounge.net>>:
>>
>>
>>     Am 24.12.2015 um 10:43 schrieb feng D:
>>
>>         I found the traffic server in my machine restart too many times
>>         in one
>>         day. It will restart 40 times in one day.
>>         I then found that traffic server has too many threads. I do not
>> know
>>         why, who can tell me.
>>
>>
>>     what is there many and why do you think that's the reason for
>> restarts?
>>     i guess you have a 6-core CPU with HT
>>
>>     normally you can say number-of-cores * 2.5 is fine which is the
>>     default as example for ffmpeg video encoding
>>
>>
>>
>> *************************************************************************************
>>         # cat /proc/1242500/task/*/comm | sort |uniq -c
>>                 1 [ACCEPT 0:8100]
>>                 3 [ET_NET 0]
>>                 2 [ET_NET 10]
>>                 2 [ET_NET 11]
>>                 4 [ET_NET 1]
>>                 7 [ET_NET 2]
>>                 7 [ET_NET 3]
>>                 8 [ET_NET 4]
>>                 5 [ET_NET 5]
>>                 5 [ET_NET 6]
>>                 3 [ET_NET 7]
>>                12 [ET_NET 8]
>>                 7 [ET_NET 9]
>>                 1 [ET_TASK 0]
>>                 1 [ET_TASK 1]
>>                 1 [LOG_FLUSH]
>>                 1 [LOG_PREPROC 0]
>>                 2 traffic_server
>>
>>
>> ****************************************************************************************
>>
>
>

Mime
View raw message