Return-Path: X-Original-To: apmail-trafficserver-users-archive@www.apache.org Delivered-To: apmail-trafficserver-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4793318673 for ; Mon, 28 Sep 2015 17:34:08 +0000 (UTC) Received: (qmail 61577 invoked by uid 500); 28 Sep 2015 17:34:08 -0000 Delivered-To: apmail-trafficserver-users-archive@trafficserver.apache.org Received: (qmail 61524 invoked by uid 500); 28 Sep 2015 17:34:07 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 61515 invoked by uid 99); 28 Sep 2015 17:34:07 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2015 17:34:07 +0000 Received: from [17.228.218.214] (unknown [17.228.218.214]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id B0AF41A0040 for ; Mon, 28 Sep 2015 17:34:06 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3095\)) Subject: Re: ats v6 crash From: James Peach X-Priority: 3 In-Reply-To: <32d5010e.dea1.15012e17ee2.Coremail.esmq@163.com> Date: Mon, 28 Sep 2015 10:34:05 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <13E29D72-028A-447A-9687-A1C4AA1D2AD8@apache.org> References: <4ad8bbcc.8059b.14ff90eaf06.Coremail.esmq@163.com> <1E2E0122-54E7-4875-B964-452DFF09E630@apache.org> <1d975f49.148fc.14fff7c31e1.Coremail.esmq@163.com> <60959831-FE8E-47ED-AF01-0F0FB9D9053B@me.com> <32d5010e.dea1.15012e17ee2.Coremail.esmq@163.com> To: users@trafficserver.apache.org X-Mailer: Apple Mail (2.3095) > On Sep 28, 2015, at 12:36 AM, Esmq wrote: >=20 > hi, i install libunwind7-dev package and recompile ats, but the = crash.log stail says: > Unable to retrieve backtrace: [12] Operation not supported on this = platform. Hmm. The requirement for backtrace support is linux, and the = libunwind-ptrace library. you can test whether this is gonna work by = doing: $ sudo traffic_ctl server backtrace this should backtrace all the traffic_server stacks (similar to = pstack(1)) >=20 > my server environment: > Process: [ET_NET 0] [4761] > File: /usr/local/trafficserver-6.0.0/bin/traffic_server > Version: Traffic Server 6.0.0 > System Version: Linux x86_64 #1 SMP Debian 3.2.65-1 3.2.0-4-amd64 > Date: Mon, 28 Sep 2015 15:24:12 +0800 >=20 > ################################## >=20 > the way i use to reproduce the crash:=20 > ab -c 10000 -n 2000000 http://x.y.com/ >=20 >=20 >=20 >=20 > =E5=9C=A8 2015-09-24 23:39:32=EF=BC=8C"James Peach" = =E5=86=99=E9=81=93=EF=BC=9A >=20 >=20 > On Sep 24, 2015, at 7:32 AM, Brian Geffon wrote: >=20 >> Unfortunately that crash log is missing the most important piece of = information because apparently backtraces aren't supported on your = platform, James, any ideas why that is?=20 >=20 > Probably no libunwind-devel at build time? >=20 >>=20 >> Are you familiar with GDB by any chance? >>=20 >> On Thursday, September 24, 2015, Esmq wrote: >> hi, the attachment is the crash.log >>=20 >> -r-------- 1 root trafficserver 87K 9=E6=9C=88 24 20:00 = crash-2015-09-24-200057.log >> -r-------- 1 root trafficserver 88K 9=E6=9C=88 24 20:01 = crash-2015-09-24-200103.log >> -r-------- 1 root trafficserver 90K 9=E6=9C=88 24 20:02 = crash-2015-09-24-200208.log >> -r-------- 1 root trafficserver 74K 9=E6=9C=88 24 20:02 = crash-2015-09-24-200215.log >> -r-------- 1 root trafficserver 73K 9=E6=9C=88 24 20:02 = crash-2015-09-24-200221.log >> -r-------- 1 root trafficserver 73K 9=E6=9C=88 24 20:02 = crash-2015-09-24-200227.log >> -r-------- 1 root trafficserver 73K 9=E6=9C=88 24 20:02 = crash-2015-09-24-200233.log >> -r-------- 1 root trafficserver 90K 9=E6=9C=88 24 20:03 = crash-2015-09-24-200344.log >> -r-------- 1 root trafficserver 73K 9=E6=9C=88 24 20:03 = crash-2015-09-24-200350.log >> -r-------- 1 root trafficserver 74K 9=E6=9C=88 24 20:03 = crash-2015-09-24-200356.log >> -r-------- 1 root trafficserver 73K 9=E6=9C=88 24 20:04 = crash-2015-09-24-200402.log >> -r-------- 1 root trafficserver 90K 9=E6=9C=88 24 20:05 = crash-2015-09-24-200524.log >> -r-------- 1 root trafficserver 87K 9=E6=9C=88 24 20:05 = crash-2015-09-24-200530.log >> -r-------- 1 root trafficserver 87K 9=E6=9C=88 24 20:05 = crash-2015-09-24-200536.log >> -r-------- 1 root trafficserver 87K 9=E6=9C=88 24 20:05 = crash-2015-09-24-200542.log >> -r-------- 1 root trafficserver 90K 9=E6=9C=88 24 20:07 = crash-2015-09-24-200723.log >> -r-------- 1 root trafficserver 73K 9=E6=9C=88 24 20:07 = crash-2015-09-24-200730.log >> -r-------- 1 root trafficserver 73K 9=E6=9C=88 24 20:07 = crash-2015-09-24-200736.log >>=20 >> At 2015-09-24 10:21:35, "Esmq" wrote: >> i will redeploy v6 on my production environment today, >> if any cores or crashlog, i'll attach here. >>=20 >> PS. crash is hard to trigger, >> and it seems like to concurrent connections. >>=20 >> At 2015-09-24 01:36:57, "James Peach" > > wrote: >> > >> >> On Sep 23, 2015, at 8:41 AM, Leif Hedstrom < >> zwoop@apache.org >> > wrote: >> >>=20 >> >>>=20 >> >>> On Sep 23, 2015, at 1:15 AM, Esmq < >> esmq@163.com >> > wrote: >> >>>=20 >> >>> hi,all >> >>> after upgrade to ats v6, i observed ats crash for several times = on all of my servers. >> >>> and it seems that when crash, it will crash again in 2 minutes. >> >>>=20 >> >>> grep enabled diags.log=20 >> >>> [Sep 14 12:02:54.705] Server {0x2b6938756700} NOTE: cache enabled >> >>> [Sep 14 12:04:05.086] Server {0x2b6b1c76c700} NOTE: cache enabled >> >>> [Sep 14 12:58:31.390] Server {0x2b13966fa700} NOTE: cache enabled >> >>> [Sep 14 12:58:37.324] Server {0x2b8681047700} NOTE: cache enabled >> >>> [Sep 14 13:00:48.937] Server {0x2ad1405be700} NOTE: cache enabled >> >>>=20 >> >>=20 >> >>=20 >> >> Did you get a core file? Also, please file a Jira. This looks like = it might be related to the new connection management features? bcall? >> > >> >Please also attach the crash log (after reviewing it for any = information you don't want made public!) >> > >> > >> >>=20 >> >> =E2=80=94 Leif >> >>=20 >> >>=20 >> >>> following is crash log in traffic.out=EF=BC=8C >> >>> = ################################################################## >> >>> traffic_server: Segmentation fault (Address not mapped to object = [0x20]) >> >>> traffic_server - STACK TRACE: >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_Z19crash_logger_invokei= P7siginfoPv+0x99)[0x4bce49] >> >>> /lib/x86_64-linux-gnu/libc.so.6(+0x321e0)[0x2af280d3d1e0] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN18UnixNetVConnection9= mainEventEiP5Event+0x30)[0x72f130] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10NetHandler9_close_v= cEP18UnixNetVConnectionlRiS2_S2_S2_+0x20f)[0x7232bf] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10NetHandler23manage_= keep_alive_queueEv+0x9d)[0x7234ad] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN17HttpClientSession7r= eleaseEP14IOBufferReader+0x1eb)[0x58b4ab] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN6HttpSM17tunnel_handl= er_uaEiP18HttpTunnelConsumer+0x265)[0x598a35] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10HttpTunnel16consume= r_handlerEiP18HttpTunnelConsumer+0x9f)[0x5e71ff] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10HttpTunnel12main_ha= ndlerEiPv+0x110)[0x5e76b0] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_Z15write_to_net_ioP10Ne= tHandlerP18UnixNetVConnectionP7EThread+0x12e1)[0x730a11] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10NetHandler12mainNet= EventEiP5Event+0x273)[0x722a83] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN7EThread7executeEv+0x= 8b0)[0x752dc0] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(main+0x14cf)[0x4a5aaf] >> >>> = /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x2af280d29ead] >> >>> /usr/local/trafficserver-6.0.0/bin/traffic_server[0x4aad1d] >> >>> traffic_server: using root directory = '/usr/local/trafficserver-6.0.0' >> >>> traffic_server: Segmentation fault (Address not mapped to object = [0x20]) >> >>> traffic_server - STACK TRACE: >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_Z19crash_logger_invokei= P7siginfoPv+0x99)[0x4bce49] >> >>> /lib/x86_64-linux-gnu/libc.so.6(+0x321e0)[0x2b6ca31a01e0] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN18UnixNetVConnection9= mainEventEiP5Event+0x30)[0x72f130] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10NetHandler9_close_v= cEP18UnixNetVConnectionlRiS2_S2_S2_+0x20f)[0x7232bf] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10NetHandler23manage_= keep_alive_queueEv+0x9d)[0x7234ad] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN17HttpClientSession7r= eleaseEP14IOBufferReader+0x1eb)[0x58b4ab] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN6HttpSM17tunnel_handl= er_uaEiP18HttpTunnelConsumer+0x265)[0x598a35] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10HttpTunnel16consume= r_handlerEiP18HttpTunnelConsumer+0x9f)[0x5e71ff] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10HttpTunnel12main_ha= ndlerEiPv+0x110)[0x5e76b0] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_Z15write_to_net_ioP10Ne= tHandlerP18UnixNetVConnectionP7EThread+0x12e1)[0x730a11] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN10NetHandler12mainNet= EventEiP5Event+0x273)[0x722a83] >> >>> = /usr/local/trafficserver-6.0.0/bin/traffic_server(_ZN7EThread7executeEv+0x= 8b0)[0x752dc0] >> >>> /usr/local/trafficserver-6.0.0/bin/traffic_server[0x75141a] >> >>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x2b6ca3500b50] >> >>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2b6ca324a70d] >> >>> traffic_server: using root directory = '/usr/local/trafficserver-6.0.0' >> > >>=20 >>=20 >>=20 >> =20 >>=20 >>=20 >> =20 >=20 >=20 > =E7=BD=91=E6=98=93=E8=80=83=E6=8B=89iPhone6s=E7=8E=AB=E7=91=B0=E9=87=915= 288=E5=85=83=EF=BC=8C=E7=8E=B0=E8=B4=A7=E4=B8=8D=E5=8A=A0=E4=BB=B7 >=20