perl-modperl mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jie Gao <J....@sydney.edu.au>
Subject Re: mod_perl Silent Failure, Very Mysterious
Date Wed, 23 Nov 2016 06:12:32 GMT
Hi Will

I can't put my finger on it, but B::Hooks::OP was last released on 11 Sept 2011, quite a while
ago, and there could be a problem with recent versions of perl.


Regards,

Jie 

* William N. Braswell, Jr. <william.braswell@autoparallel.com> wrote:

> Date: Tue, 22 Nov 2016 23:14:11 -0600
> From: "William N. Braswell, Jr." <william.braswell@autoparallel.com>
> To: Jie Gao <j.gao@sydney.edu.au>
> CC: modperl@perl.apache.org
> Subject: Re: mod_perl Silent Failure, Very Mysterious
> 
> Hi Jie,
> If I temporarily disable B::Hooks::OP::Check, as shown below, then it
> tells me it was included from Devel::Declare, which was in turn
> (indireclty) included from Method::Signatures::Simple.
> I am not the author of either Catalyst or ShinyCMS, so unfortunately I
> don't know any more in-depth details off the top of my head.
> Does that answer your question?
> Thanks,~ Will
> wbraswell@cloud-comp0-00:~$ pm_location.sh
> B::Hooks::OP::Check/home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/B/Hooks/OP/Check.pm
> wbraswell@cloud-comp0-00:~$ mv
> /home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/B/Hooks/OP/Check.pm
> /home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/B/Hooks/OP/Check.pm.DISABLED
> wbraswell@cloud-comp0-00:~/public_html/cloud-comp0-00.autoparallel.com-latest$
> perl -e 'use ShinyCMS;'Can't locate B/Hooks/OP/Check.pm in @INC (you
> may need to install the B::Hooks::OP::Check module) (@INC contains
> ....) at
> /home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Devel/Declare.pm
> line 18.BEGIN failed--compilation aborted at
> /home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Devel/Declare.pm
> line 18.Compilation failed in require at
> /home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Devel/Declare/Context/Simple.pm
> line 5.BEGIN failed--compilation aborted at
> /home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Devel/Declare/Context/Simple.pm
> line 5.Compilation failed in require at /usr/share/perl/5.22/base.pm
> line 97. ...propagated at /usr/share/perl/5.22/base.pm line 106.BEGIN
> failed--compilation aborted at
> /home/wbraswell/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Devel/Declare/MethodInstaller/Simple.pm
> line 3.Compilation failed in require at /usr/share/perl/5.22/base.pm
> line 97. ...propagated at /usr/share/perl/5.22/base.pm line 106.BEGIN
> failed--compilation aborted at
> /home/wbraswell/perl5/lib/perl5/Method/Signatures/Simple.pm line
> 19.Compilation failed in require at blib/lib/ShinyCMS.pm line 21.BEGIN
> failed--compilation aborted at blib/lib/ShinyCMS.pm line
> 21.Compilation failed in require at -e line 1.BEGIN
> failed--compilation aborted at -e line 1.
> On 11/22/2016 at 11:03 PM, "Jie Gao"  wrote:Hi William
> 
> Where excatly is B::Hooks::OP::Check required / its Check.xs called?
> Regards,
> Jie 
> 
> * William N. Braswell, Jr.  wrote:
> 
> > Date: Tue, 22 Nov 2016 06:04:36 -0600
> > From: "William N. Braswell, Jr." 
> > To: Jie Gao 
> > CC: modperl@perl.apache.org
> > Subject: RE: mod_perl Silent Failure, Very Mysterious
> > 
> > Hello Again Jie,
> > I have run the modperl_testreport.pl script per your direction, and
> > have attached the output file.
> > Also, I have posted this as a mod_perl issue:
> > 
> > https://rt.cpan.org/Ticket/Display.html?id=118900
> > I am attempting to run an already-bug-free Catalyst application
> under
> > mod_perl, which should work just fine but instead gives a segfault
> in
> > the Check.xs file.
> > The name of the Catalyst application is ShinyCMS, it works just fine
> > using the PSGI stand-alone testing web server as well as fastcgi,
> and
> > mod_perl is the only option which causes a segfault.  I must have
> > mod_perl support for my specific use of ShinyCMS, so I really really
> > need to fix this segfault and get Shiny working under mod_perl. 
> Yes,
> > Shiny is specifically designed to work with mod_perl, as are most
> > Catalyst apps.  In fact, my simple ">" print statements in
> ShinyCMS.pm
> > show that Shiny is totally loading and returning control back to
> > Apache (or mod_perl?) before the actual segfault occurs.
> > My system info:
> > $ apache2 -vServer version: Apache/2.4.18 (Ubuntu)Server built:  
> > 2016-07-14T12:32:26
> > $ perl -vThis is perl 5, version 22, subversion 1 (v5.22.1) built
> for
> > x86_64-linux-gnu-thread-multi(with 58 registered patches, see perl
> -V
> > for more detail)
> > $ pm_version.pl mod_perl2$mod_perl2::VERSION = 2.000009
> > $ pm_version.pl B::Hooks::OP::Check$B::Hooks::OP::Check::VERSION =
> > 0.19
> > $ cat /etc/issueUbuntu 16.04.1 LTS
> > I have spent the last week trying all of the following and more, but
> > the segfault persists:
> > 1.  manually-compiled perl v5.22.1 & mod_perl2 v2.000009 w/out
> > threads2.  mod_perl2 back to v2.0000053.  B::Hooks::OP::Check back
> to
> > v0.17 (maybe earlier)4.  move all possible 'use' statements to be
> > called as early as possible in Shiny_dependencies.pm
> > I also tried Apache v2.2 & mod_perl2 v2.000008 & Perl v5.18.2 in
> > Ubuntu v14.04, it did not segfault but instead it served up totally
> > blank pages with 0 bytes of data.  I have not dug deeper because I
> use
> > Ubuntu v16.04 in production and I figure it is harder to debug w/out
> > an actual segfault to start from.
> > I have set up SSH access on a cloud server so you can see and debug
> > the segfault directly, all 3 of the usernames below are set as
> sudoers
> > so please wield your root powers with caution:
> > HOSTNAME: cloud-comp0-00.autoparallel.comHOST IP: 
> > 64.137.255.53USERNAME: shay OR fmoyer OR jgao OR
> modperl_teamPASSWORD:
> > check your CPAN e-mail address
> > Pertinent paths:
> >
> /home/wbraswell/public_html/cloud-comp0-00.autoparallel.com-latest/etc/apache2/sites-available/cloud-comp0-00.autoparallel.com.conf
> > Here is an example gdb showing Check.xs as the likely culprit,
> > starting with `sudo` and then `source` to load Apache env vars:
> > [[[ BEGIN GDB SESSION ]]]
> > zefram@cloud-comp0-00:/$ sudo -i...root@cloud-comp0-00:~# source
> > /etc/apache2/envvarsroot@cloud-comp0-00:~# gdb
> > /usr/sbin/apache2...Reading symbols from /usr/sbin/apache2...(no
> > debugging symbols found)...done.
> > (gdb) run -k start -XStarting program: /usr/sbin/apache2 -k start
> > -X[Thread debugging using libthread_db enabled]Using host
> libthread_db
> > library "/lib/x86_64-linux-gnu/libthread_db.so.1".>: top of
> > ShinyCMS.pm>: in ShinyCMS.pm, about to use Moose>: in ShinyCMS.pm,
> > about to use Catalyst::Runtime>: in ShinyCMS.pm, about to use
> > Catalyst>: in ShinyCMS.pm, about to use CatalystX::RoleApplicator>:
> in
> > ShinyCMS.pm, about to use Method::Signatures::Simple>: in
> ShinyCMS.pm,
> > about to call config()>: in ShinyCMS.pm, have __PACKAGE__ =
> ShinyCMS>:
> > in ShinyCMS.pm, about to call setup()...>: in ShinyCMS.pm, returned
> > from setup()>: in ShinyCMS.pm, about to return 1
> > Program received signal SIGSEGV, Segmentation
> fault.0x00007fffebabcb10
> > in ?? ()(gdb) bt#0  0x00007fffebabcb10 in ?? ()#1 
> 0x00007ffff3e5302b
> > in Perl_newUNOP (my_perl=my_perl@entry=0x5555577ddde0,
> > type=type@entry=17, flags=, flags@entry=8192, first=0x5555562141b8)
> at
> > op.c:4811#2  0x00007ffff3e54a1d in Perl_newCVREF
> > (my_perl=my_perl@entry=0x5555577ddde0, flags=flags@entry=8192, o=)
> at
> > op.c:9367#3  0x00007ffff3e8b686 in Perl_yylex
> > (my_perl=my_perl@entry=0x5555577ddde0) at toke.c:6693#4 
> > 0x00007ffff3e97228 in Perl_yyparse
> > (my_perl=my_perl@entry=0x5555577ddde0, gramtype=gramtype@entry=258)
> at
> > perly.c:322...#44 0x000055555558709f in main ()
> > (gdb) startThe program being debugged has been started already.Start
> > it from the beginning? (y or n) yTemporary breakpoint 1 at
> > 0x5555555867b0Starting program: /usr/sbin/apache2 -k start -X[Thread
> > debugging using libthread_db enabled]Using host libthread_db library
> > "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > Temporary breakpoint 1, 0x00005555555867b0 in main ()
> > (gdb) break perl_parseFunction "perl_parse" not defined.Make
> > breakpoint pending on future shared library load? (y or [n])
> > yBreakpoint 2 (perl_parse) pending.
> > (gdb) continueContinuing.
> > Breakpoint 2, perl_parse (my_perl=0x55555581ae50,
> > xsinit=0x7ffff420a280, argc=2, argv=0x7ffff7f88070, env=0x0) at
> > perl.c:14731473    perl.c: No such file or directory.
> > (gdb) watch -l PL_check[17]Hardware watchpoint 3: -location
> > PL_check[17]
> > (gdb) continueContinuing.
> > Hardware watchpoint 3: -location PL_check[17]
> > Old value = (OP *(*)(PerlInterpreter *, OP *)) 0x7ffff3e4d9e0 New
> > value = (OP *(*)(PerlInterpreter *, OP *)) 0x7fffebabcb10
> > 0x00007fffebabcd0e in hook_op_check (type=type@entry=OP_RV2CV,
> > cb=cb@entry=0x7fffeb8b6d30 , user_data=user_data@entry=0x0) at
> > Check.xs:6666                      PL_check[type] = check_cb;
> > (gdb) info threads  Id   Target Id         Frame * 1    Thread
> > 0x7ffff7fe2780 (LWP 6196) "/usr/sbin/apach" 0x00007fffebabcd0e in
> > hook_op_check (type=type@entry=OP_RV2CV, cb=cb@entry=0x7fffeb8b6d30
> ,
> > user_data=user_data@entry=0x0)    at Check.xs:66
> > (gdb) info sourceCurrent source file is Check.xsCompilation
> directory
> > is
> >
> /home/wbraswell/.cpanm/work/1479735542.24787/B-Hooks-OP-Check-0.19Located
> > in
> >
> /home/wbraswell/.cpanm/work/1479735542.24787/B-Hooks-OP-Check-0.19/Check.xsContains
> > 106 lines.Source language is c.Producer is GNU C11 5.4.0 20160609
> > -mtune=generic -march=x86-64 -g -O2 -fwrapv -fno-strict-aliasing
> -fPIC
> > -fstack-protector-strong.Compiled with DWARF 2 debugging format.Does
> > not include preprocessor macro info.
> > (gdb) continueContinuing.>: top of ShinyCMS.pm>: in ShinyCMS.pm,
> about
> > to use Moose>: in ShinyCMS.pm, about to use Catalyst::Runtime>: in
> > ShinyCMS.pm, about to use Catalyst>: in ShinyCMS.pm, about to use
> > CatalystX::RoleApplicator>: in ShinyCMS.pm, about to use
> > Method::Signatures::Simple>: in ShinyCMS.pm, about to call
> config()>:
> > in ShinyCMS.pm, have __PACKAGE__ = ShinyCMS>: in ShinyCMS.pm, about
> to
> > call setup()...>: in ShinyCMS.pm, returned from setup()>: in
> > ShinyCMS.pm, about to return 1
> > ...
> > Program received signal SIGSEGV, Segmentation
> fault.0x00007fffebabcb10
> > in ?? ()(gdb) bt#0  0x00007fffebabcb10 in ?? ()#1 
> 0x00007ffff3e5302b
> > in Perl_newUNOP (my_perl=my_perl@entry=0x555559dd8ed0,
> > type=type@entry=17, flags=, flags@entry=8192, first=0x55555824eb98)
> at
> > op.c:4811#2  0x00007ffff3e54a1d in Perl_newCVREF
> > (my_perl=my_perl@entry=0x555559dd8ed0, flags=flags@entry=8192, o=)
> at
> > op.c:9367...#44 0x000055555558709f in main ()
> > (gdb) print PL_check[17]$1 = (Perl_check_t) 0x7fffebabcb10
> > (gdb) quit
> > [[[ END GDB SESSION ]]]
> > I have attached the unabbreviated gdb session and the system info in
> > text files, for your reference.
> > I am anxious (near desperate) to get this bug fixed so that I can
> move
> > forward with my work.  I will eagerly assist in any way I can,
> please
> > just let me know what I should do and I will do it!
> > Thanks so much for your help.
> > Perling,~ Will
> > 
> > On 11/16/2016 at 3:17 AM, "Jie Gao"  wrote:     For the completeness
> > of issue reporting, please run the following script and post the
> > output:
> >  #!/usr/bin/perl
> >  use strict;
> >  use warnings FATAL => 'all';
> >  use ModPerl::TestReport;
> >  ModPerl::TestReport->new(@ARGV)->run;
> >  Jie
> > -------------------------
> >  From: William N. Braswell, Jr. [william.braswell@autoparallel.com]
> >  Sent: Tuesday, November 15, 2016 8:01 PM
> >  To: Jie Gao
> >  Cc: modperl@perl.apache.org
> >  Subject: Re: mod_perl Silent Failure, Very Mysterious
> >     Hello Jie & Friends, 
> >   I got the core dump out of gdb and also some hopefully-helpful
> > output. 
> >   Here is the output from gdb (not the core dump which is binary of
> > course): 
> >   http://hastebin.com/okaxacuriq.scala
> >  Then again maybe we have a problem because I'm missing debugging
> > symbols inside /usr/sbin/apache2?
> >  _Program received signal SIGSEGV, Segmentation fault.   
> > 0x00007fffeed67a00 in ?? ()_
> >  Note the _??_ part in the output above... 
> >   Where do I go from here? 
> >   Thanks, ~ Will 
> >  On 11/15/2016 at 12:14 AM, "William N. Braswell, Jr."  wrote:   Hi
> > Jie, 
> >   I have captured the output of `strace -f /usr/sbin/apache2 -e
> trace8
> > -k start`: 
> >   http://hastebin.com/unupocozop.pl 
> >   Looks like it is dying with a segfault, but I'm not sure what is
> > causing this??? 
> >   Thanks so much for your help!!!  :-) 
> >   ~ Will 
> >   [[[ BEGIN PASTE SNIPPET ]]] 
> >    [pid  6550] stat("blib/lib/DynaLoader.pmc", 0x7ffdda87f280) = -1
> > ENOENT (No such file or directory) [pid  6550]
> > stat("blib/lib/DynaLoader.pm", 0x7ffdda87f1b0) = -1 ENOENT (No such
> > file or directory) [pid  6550] stat("/etc/perl/DynaLoader.pmc",
> > 0x7ffdda87f280) = -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/etc/perl/DynaLoader.pm", 0x7ffdda87f1b0) = -1 ENOENT (No such
> > file or directory) [pid  6550]
> > stat("/usr/local/lib/x86_64-linux-gnu/perl/5.22.1/DynaLoader.pmc",
> > 0x7ffdda87f280) = -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/usr/local/lib/x86_64-linux-gnu/perl/5.22.1/DynaLoader.pm",
> > 0x7ffdda87f1b0) = -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/usr/local/share/perl/5.22.1/DynaLoader.pmc", 0x7ffdda87f280)
> =
> > -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/usr/local/share/perl/5.22.1/DynaLoader.pm", 0x7ffdda87f1b0) =
> > -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/usr/lib/x86_64-linux-gnu/perl5/5.22/DynaLoader.pmc",
> > 0x7ffdda87f280) = -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/usr/lib/x86_64-linux-gnu/perl5/5.22/DynaLoader.pm",
> > 0x7ffdda87f1b0) = -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/usr/share/perl5/DynaLoader.pmc", 0x7ffdda87f280) = -1 ENOENT
> > (No such file or directory) [pid  6550]
> > stat("/usr/share/perl5/DynaLoader.pm", 0x7ffdda87f1b0) = -1 ENOENT
> (No
> > such file or directory) [pid  6550]
> > stat("/usr/lib/x86_64-linux-gnu/perl/5.22/DynaLoader.pmc",
> > 0x7ffdda87f280) = -1 ENOENT (No such file or directory) [pid  6550]
> > stat("/usr/lib/x86_64-linux-gnu/perl/5.22/DynaLoader.pm",
> > {st_mode=S_IFREG|0644, st_size=10748, ...}) = 0 [pid  6550]
> > open("/usr/lib/x86_64-linux-gnu/perl/5.22/DynaLoader.pm", O_RDONLY)
> =
> > 5 [pid  6550] ioctl(5, TCGETS, 0x7ffdda87ef50) = -1 ENOTTY
> > (Inappropriate ioctl for device) [pid  6550] lseek(5, 0, SEEK_CUR)  
>  
> >   = 0 [pid  6550] read(5, "# Generated from DynaLoader_pm.P"...,
> 8192)
> > = 8192 [pid  6550] --- SIGSEGV {si_signo=SIGSEGV,
> si_code=SEGV_MAPERR,
> > si_addr=0x7f491064fa00} --- [pid  6550] chdir("/etc/apache2")      
> =
> > 0 [pid  6550] rt_sigaction(SIGSEGV, {SIG_DFL, [],
> > SA_RESTORER|SA_INTERRUPT, 0x7f4918dea3d0}, {SIG_DFL, [],
> > SA_RESTORER|SA_RESETHAND, 0x7f4918dea3d0}, 8) = 0 [pid  6550]
> > kill(6550, SIGSEGV)         = 0 [pid  6550] rt_sigreturn({mask=[]}) 
>  
> >  = 139951873083648 [pid  6550] --- SIGSEGV {si_signo=SIGSEGV,
> > si_code=SI_USER, si_pid=6550, si_uid=0} --- [pid  6549] +++ exited
> > with 0 +++ +++ killed by SIGSEGV (core dumped) +++  
> >   [[[ END PASTE SNIPPET ]]] 
> >  On 11/14/2016 at 11:54 PM, "Jie Gao"  wrote:  Add the "-f" flag to
> > strace and see what errors you get.
> >  Regards,
> >  Jie 
> >  * William N. Braswell, Jr.  wrote:
> >  > Date: Mon, 14 Nov 2016 23:27:53 -0600
> >  > From: "William N. Braswell, Jr." 
> >  > To: modperl@perl.apache.org
> >  > Subject: mod_perl Silent Failure, Very Mysterious
> >  > 
> >  > Hello everyone, I'm trying to enable a web app (Catalyst
> ShinyCMS)
> > to
> >  > run under mod_perl, it is definitely supposed to work but it
> fails
> >  > without any errors, and I've been trying for several days to get
> it
> >  > working.
> >  > My system info:
> >  > * Perl 5, version 22, subversion 1 (v5.22.1) built for
> >  > x86_64-linux-gnu-thread-multi
> >  > * libapache2-mod-perl2 v2.0.9-4ubuntu1
> >  > * Apache/2.4.18 (Ubuntu)
> >  > * $Apache::Test::VERSION = 1.39
> >  > * $Catalyst::VERSION = 5.90112
> >  > * $ShinyCMS::VERSION = 0.007
> >  > I know the app already functions properly when running via the
> >  > stand-alone test server script (PSGI) or fastcgi, and I am able
> to
> > run
> >  > a trivial Catalyst test application successfully under mod_perl,
> so
> >  > this issue seems to have something to do with the interaction
> > between
> >  > the ShinyCMS app and mod_perl.
> >  > I have added print debug statements in my app, so when I call my
> > app
> >  > from within the Apache site conf file then I can see the debug
> >  > statements prove that it is successfully going all the way
> through
> > my
> >  > app, and it even returns control to the embedded code w/in the
> > Apache
> >  > site conf file, but it still just silently fails anyway with the
> >  > following non-descriptive error:
> >  > "The apache2 instance did not start within 20 seconds. Please
> read
> > the
> >  > log files to discover problems"
> >  > Unfortunately the log files do not provide any additional
> > information
> >  > whatsoever!
> >  > Here are my Apache config files and the output of trying to start
> >  > Apache manually, you can see my ">" statements which prove that
> my
> >  > ShinyCMS app code is definitely executing all the way through and
> >  > returning control to Apache:
> >  > http://hastebin.com/rulixitole.php
> >  > http://hastebin.com/wevicagano.rb
> >  > This is a really mysterious issue, when I run `apache2ctl -e
> debug
> > -k
> >  > start` as seen in the link above, it goes through the motions as
> if
> > it
> >  > is starting correctly, and it does not give any error messages,
> but
> >  > when apache2ctl is done there simply are not any apache processes
> >  > running...
> >  > I only found 1 person on google with a possibly-similar issue,
> but
> > he
> >  > told me that he never found a solution:
> >  > https://github.com/timbunce/devel-nytprof/issues/89
> >  > I also found 1 person on the mod_perl mailing list archives with
> >  > another possibly-similar issue, but I have no way to follow up,
> and
> >  > I'm not sure of the best way to totally rebuild everything from
> >  > source:
> >  > 
> >
> https://mail-archives.apache.org/mod_mbox/perl-modperl/200207.mbox/%3C4A19F19A1BB4D51183220020ED069714012882B0@exch2.iso-ne.com%3E
> >  > I have asked for help on IRC #httpd, user Unbeliever suggested I
> >  > ascertain my real /usr/sbin/apache2 command line and run it
> through
> >  > strace, then post to this mailing list.
> >  > I used -x in apache2ctl which showed me I was simply sourcing
> >  > /tmp/apache2/envvars and then running `/usr/sbin/apache2 -e
> trace8
> > -k
> >  > start` which are the exact same args as accepted by the
> apache2ctl
> >  > script.
> >  > I have now run `strace /usr/sbin/apache2 -e trace8 -k start`
> which
> >  > generates 2.5MB of text output, I have pasted the last few pages
> of
> > it
> >  > on hastebin, I can post the whole thing for download if needed:
> >  > http://hastebin.com/idemogexiq.erl
> >  > I am not sure how to interpret the results of the strace command
> >  > output and debug this issue, help please?
> >  > Any and all assistance is greatly appreciated! :-)
> >  > Thanks,~ Will

Mime
View raw message