Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (athena.apache.org: domain of thomas.r.w.eckert@gmail.com
 designates 209.85.217.182 as permitted sender)
MIME-Version: 1.0
Date: Fri, 24 May 2013 10:38:19 +0200
Message-ID: 
 <CAPV0b04sECKVG=GkTtb-+7MiCD-Seuym51iyc+Kgxm4t+nmw0A@mail.gmail.com>
Subject: mod_security core dumps and r->per_dir_config
From: Thomas Eckert <thomas.r.w.eckert@gmail.com>
To: dev@httpd.apache.org
Content-Type: multipart/alternative; boundary=001a11c34b4a9b958e04dd72b869

--001a11c34b4a9b958e04dd72b869
Content-Type: text/plain; charset=ISO-8859-1

I'm trying to investigate some core dumps in mod_security and currently
face this

(gdb) bt
#0  0xf6efc232 in create_tx_context (r=0x1eac8ed0) at mod_security2.c:325
#1  0xf6efc606 in hook_error_log (file=0x80a51bd "http_filters.c",
line=493, level=3, status=104, s=0x18144178, r=0x1eac8ed0, mp=0x0,
fmt=0xeb2f6dff "Error reading chunk  \n")
    at mod_security2.c:771
#2  0x08080c2d in ap_run_error_log (file=0x80a51bd "http_filters.c",
line=493, level=3, status=104, s=0x18144178, r=0x1eac8ed0, pool=0x0,
    errstr=0xeb2f6dff "Error reading chunk  \n") at log.c:1116
#3  0x0808109b in log_error_core (file=0x80a51bd "http_filters.c",
line=493, level=<optimized out>, status=104, s=0x18144178, c=0x0,
r=0x1eac8ed0, pool=0x0,
    fmt=0x80a52a1 "Error reading chunk %s ", args=0xeb2f8e08
"hT\n\b\230L\255\036\177\017\275\036\376\001") at log.c:705
#4  0x08081ec1 in ap_log_rerror (file=0x80a51bd "http_filters.c", line=493,
level=3, status=104, r=0x1eac8ed0, fmt=0x80a52a1 "Error reading chunk %s ")
at log.c:737
#5  0x0808d400 in ap_http_filter (f=0x1eaca3b0, b=0x1eac8eb0,
mode=AP_MODE_READBYTES, block=APR_NONBLOCK_READ, readbytes=8192) at
http_filters.c:493
#6  0xf716bfe6 in ap_proxy_http_process_response (p=0x1eab4cd8,
r=0x1eab4d18, backend=0x9241848, origin=0x929ab40, conf=0x1e8db5e0,
server_portstr=0xeb2fd107 "")
    at mod_proxy_http.c:1771
#7  0xf716d609 in proxy_http_handler (r=0x1eab4d18, worker=0x13db7be8,
conf=0x1e8db5e0, url=0x1eac8950 "/a/b/WebService", proxyname=0x0,
    proxyport=<optimized out>) at mod_proxy_http.c:2037
#8  0xf72daec0 in proxy_run_scheme_handler (r=0x1eab4d18,
worker=0x13db7be8, conf=0x1e8db5e0, url=0x1eac8878 "
http://10.10.10.10:8080/a/b/WebService<http://10.21.9.205:8080/tourconex/services/TourConexWebService>",

    proxyhost=0x0, proxyport=0) at mod_proxy.c:2455
#9  0xf72dfdbb in proxy_handler (r=0x1eab4d18) at mod_proxy.c:1023
#10 0x0807cdc1 in ap_run_handler (r=0x1eab4d18) at config.c:157
#11 0x080802c6 in ap_invoke_handler (r=0x1eab4d18) at config.c:376
#12 0x0808b6d0 in ap_process_request (r=0x1eab4d18) at http_request.c:282
#13 0x080886e8 in ap_process_http_connection (c=0x1eaaaea8) at
http_core.c:190
#14 0x08084571 in ap_run_process_connection (c=0x1eaaaea8) at
connection.c:43
#15 0x08091aac in process_socket (bucket_alloc=<optimized out>,
my_thread_num=<optimized out>, my_child_num=<optimized out>,
sock=<optimized out>, p=<optimized out>)
    at worker.c:544
#16 worker_thread (thd=0x1e96ac40, dummy=0x1ea48678) at worker.c:894
#17 0xf7575ac6 in dummy_worker (opaque=0x1e96ac40) at
threadproc/unix/thread.c:142
#18 0xf74fc809 in start_thread () from /lib/libpthread.so.0
#19 0xf745d30e in clone () from /lib/libc.so.6
Backtrace stopped: Not enough registers or memory available to unwind
further
(gdb) print r->per_dir_config
$1 = (struct ap_conf_vector_t *) 0x0

where the offending line of code is

msr->dcfg1 = (directory_config *)ap_get_module_config(r->per_dir_config,
&security2_module);

Why would the per_dir_config be NULL here ? I don't think that should ever
be encountered during the request's lifetime, right ?

My confusion continued when I saw

(gdb) print *r
$2 = {pool = 0x1eab4cd8, connection = 0x929ab40, server = 0x18144178, next
= 0x0, prev = 0x0, main = 0x0, the_request = 0x0, assbackwards = 0,
proxyreq = 3, header_only = 0,
  protocol = 0x0, proto_num = 0, hostname = 0x0, request_time =
1368183918343107, status_line = 0x0, status = 200, method = 0x0,
method_number = 0, allowed = 0,
  allowed_xmethods = 0x0, allowed_methods = 0x0, sent_bodyct = 0,
bytes_sent = 0, mtime = 0, chunked = 0, range = 0x0, clength = 0, remaining
= 0, read_length = 0,
  read_body = 0, read_chunked = 0, expecting_100 = 0, headers_in =
0x1ebcee40, headers_out = 0x1eac9750, err_headers_out = 0x1eac98f8,
subprocess_env = 0x1eac93e0,
  notes = 0x1eac9a50, content_type = 0x0, handler = 0x0, content_encoding =
0x0, content_languages = 0x0, vlist_validator = 0x0, user = 0x0,
ap_auth_type = 0x0, no_cache = 0,
  no_local_copy = 0, unparsed_uri = 0x0, uri = 0x0, filename = 0x0,
canonical_filename = 0x0, path_info = 0x0, args = 0x0, finfo = {pool = 0x0,
valid = 0, protection = 0,
    filetype = APR_NOFILE, user = 0, group = 0, inode = 0, device = 0,
nlink = 0, size = 0, csize = 0, atime = 0, mtime = 0, ctime = 0, fname =
0x0, name = 0x0,
    filehand = 0x0}, parsed_uri = {scheme = 0x0, hostinfo = 0x0, user =
0x0, password = 0x0, hostname = 0x0, port_str = 0x0, path = 0x0, query =
0x0, fragment = 0x0,
    hostent = 0x0, port = 0, is_initialized = 0, dns_looked_up = 0,
dns_resolved = 0}, used_path_info = 0, per_dir_config = 0x0, request_config
= 0x1eac9ba8, htaccess = 0x0,
  output_filters = 0x929aff8, input_filters = 0x1eaca3b0,
proto_output_filters = 0x929aff8, proto_input_filters = 0x1eaca3b0,
eos_sent = 0}
(gdb) dump_table(r->headers_in)
[0] 'Server'='Apache-Coyote/1.1' [0x1eaca1e0]
[1] 'X-Powered-By'='Servlet 2.4; JBoss-4.2.2.GA (build:
SVNTag=JBoss_4_2_2_GA date=200710221139)/Tomcat-5.5' [0x1eaca218]
[2] 'Content-Type'='text/xml;charset=utf-8' [0x1eaca290]
[3] 'Transfer-Encoding'='chunked' [0x1eaca2d0]
[4] 'Date'='Fri, 10 May 2013 11:05:59 GMT' [0x1eaca310]

A 'Server' header in the headers_in field, isn't that just nonsense ? Also
note the NULLs in for protocol and proto_num which I don't think should be
there.

I got the feeling something was really wrong with that request, long before
mod_security got to see it. Unfortunately, I don't know the circumstances
of the request since I only have access to the coredump file and nothing
else.

Any hint on how to proceed with this, e.g. what to inspect ? I don't think
it's only per_dir_config screwed up here.

Cheers & thanks,
  Thomas

--001a11c34b4a9b958e04dd72b869
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>I&#39;m trying to investigate some core dumps in=
 mod_security and currently face this<br><br>(gdb) bt<br>#0=A0 0xf6efc232 i=
n create_tx_context (r=3D0x1eac8ed0) at mod_security2.c:325<br>#1=A0 0xf6ef=
c606 in hook_error_log (file=3D0x80a51bd &quot;http_filters.c&quot;, line=
=3D493, level=3D3, status=3D104, s=3D0x18144178, r=3D0x1eac8ed0, mp=3D0x0, =
fmt=3D0xeb2f6dff &quot;Error reading chunk=A0 \n&quot;)<br>

=A0=A0=A0 at mod_security2.c:771<br>#2=A0 0x08080c2d in ap_run_error_log (f=
ile=3D0x80a51bd &quot;http_filters.c&quot;, line=3D493, level=3D3, status=
=3D104, s=3D0x18144178, r=3D0x1eac8ed0, pool=3D0x0, <br>=A0=A0=A0 errstr=3D=
0xeb2f6dff &quot;Error reading chunk=A0 \n&quot;) at log.c:1116<br>

#3=A0 0x0808109b in log_error_core (file=3D0x80a51bd &quot;http_filters.c&q=
uot;, line=3D493, level=3D&lt;optimized out&gt;, status=3D104, s=3D0x181441=
78, c=3D0x0, r=3D0x1eac8ed0, pool=3D0x0, <br>=A0=A0=A0 fmt=3D0x80a52a1 &quo=
t;Error reading chunk %s &quot;, args=3D0xeb2f8e08 &quot;hT\n\b\230L\255\03=
6\177\017\275\036\376\001&quot;) at log.c:705<br>

#4=A0 0x08081ec1 in ap_log_rerror (file=3D0x80a51bd &quot;http_filters.c&qu=
ot;, line=3D493, level=3D3, status=3D104, r=3D0x1eac8ed0, fmt=3D0x80a52a1 &=
quot;Error reading chunk %s &quot;) at log.c:737<br>#5=A0 0x0808d400 in ap_=
http_filter (f=3D0x1eaca3b0, b=3D0x1eac8eb0, mode=3DAP_MODE_READBYTES, bloc=
k=3DAPR_NONBLOCK_READ, readbytes=3D8192) at http_filters.c:493<br>

#6=A0 0xf716bfe6 in ap_proxy_http_process_response (p=3D0x1eab4cd8, r=3D0x1=
eab4d18, backend=3D0x9241848, origin=3D0x929ab40, conf=3D0x1e8db5e0, server=
_portstr=3D0xeb2fd107 &quot;&quot;)<br>=A0=A0=A0 at mod_proxy_http.c:1771<b=
r>#7=A0 0xf716d609 in proxy_http_handler (r=3D0x1eab4d18, worker=3D0x13db7b=
e8, conf=3D0x1e8db5e0, url=3D0x1eac8950 &quot;/a/b/WebService&quot;, proxyn=
ame=3D0x0, <br>

=A0=A0=A0 proxyport=3D&lt;optimized out&gt;) at mod_proxy_http.c:2037<br>#8=
=A0 0xf72daec0 in proxy_run_scheme_handler (r=3D0x1eab4d18, worker=3D0x13db=
7be8, conf=3D0x1e8db5e0, url=3D0x1eac8878 &quot;<a href=3D"http://10.21.9.2=
05:8080/tourconex/services/TourConexWebService" target=3D"_blank">http://10=
.10.10.10:8080/a/b/WebService</a>&quot;, <br>

=A0=A0=A0 proxyhost=3D0x0, proxyport=3D0) at mod_proxy.c:2455<br>#9=A0 0xf7=
2dfdbb in proxy_handler (r=3D0x1eab4d18) at mod_proxy.c:1023<br>#10 0x0807c=
dc1 in ap_run_handler (r=3D0x1eab4d18) at config.c:157<br>#11 0x080802c6 in=
 ap_invoke_handler (r=3D0x1eab4d18) at config.c:376<br>

#12 0x0808b6d0 in ap_process_request (r=3D0x1eab4d18) at http_request.c:282=
<br>#13 0x080886e8 in ap_process_http_connection (c=3D0x1eaaaea8) at http_c=
ore.c:190<br>#14 0x08084571 in ap_run_process_connection (c=3D0x1eaaaea8) a=
t connection.c:43<br>

#15 0x08091aac in process_socket (bucket_alloc=3D&lt;optimized out&gt;, my_=
thread_num=3D&lt;optimized out&gt;, my_child_num=3D&lt;optimized out&gt;, s=
ock=3D&lt;optimized out&gt;, p=3D&lt;optimized out&gt;)<br>=A0=A0=A0 at wor=
ker.c:544<br>

#16 worker_thread (thd=3D0x1e96ac40, dummy=3D0x1ea48678) at worker.c:894<br=
>#17 0xf7575ac6 in dummy_worker (opaque=3D0x1e96ac40) at threadproc/unix/th=
read.c:142<br>#18 0xf74fc809 in start_thread () from /lib/libpthread.so.0<b=
r>

#19 0xf745d30e in clone () from /lib/libc.so.6<br>Backtrace stopped: Not en=
ough registers or memory available to unwind further<br>(gdb) print r-&gt;p=
er_dir_config<br>$1 =3D (struct ap_conf_vector_t *) 0x0<br><br></div>where =
the offending line of code is<br>

<br>msr-&gt;dcfg1 =3D (directory_config *)ap_get_module_config(r-&gt;per_di=
r_config, &amp;security2_module);<br><br></div><div>Why would the per_dir_c=
onfig be NULL here ? I don&#39;t think that should ever be encountered duri=
ng the request&#39;s lifetime, right ?<br>

<br></div><div>My confusion continued when I saw<br><br>(gdb) print *r<br>$=
2 =3D {pool =3D 0x1eab4cd8, connection =3D 0x929ab40, server =3D 0x18144178=
, next =3D 0x0, prev =3D 0x0, main =3D 0x0, the_request =3D 0x0, assbackwar=
ds =3D 0, proxyreq =3D 3, header_only =3D 0, <br>

=A0 protocol =3D 0x0, proto_num =3D 0, hostname =3D 0x0, request_time =3D 1=
368183918343107, status_line =3D 0x0, status =3D 200, method =3D 0x0, metho=
d_number =3D 0, allowed =3D 0, <br>=A0 allowed_xmethods =3D 0x0, allowed_me=
thods =3D 0x0, sent_bodyct =3D 0, bytes_sent =3D 0, mtime =3D 0, chunked =
=3D 0, range =3D 0x0, clength =3D 0, remaining =3D 0, read_length =3D 0, <b=
r>

=A0 read_body =3D 0, read_chunked =3D 0, expecting_100 =3D 0, headers_in =
=3D 0x1ebcee40, headers_out =3D 0x1eac9750, err_headers_out =3D 0x1eac98f8,=
 subprocess_env =3D 0x1eac93e0, <br>=A0 notes =3D 0x1eac9a50, content_type =
=3D 0x0, handler =3D 0x0, content_encoding =3D 0x0, content_languages =3D 0=
x0, vlist_validator =3D 0x0, user =3D 0x0, ap_auth_type =3D 0x0, no_cache =
=3D 0, <br>

=A0 no_local_copy =3D 0, unparsed_uri =3D 0x0, uri =3D 0x0, filename =3D 0x=
0, canonical_filename =3D 0x0, path_info =3D 0x0, args =3D 0x0, finfo =3D {=
pool =3D 0x0, valid =3D 0, protection =3D 0, <br>=A0=A0=A0 filetype =3D APR=
_NOFILE, user =3D 0, group =3D 0, inode =3D 0, device =3D 0, nlink =3D 0, s=
ize =3D 0, csize =3D 0, atime =3D 0, mtime =3D 0, ctime =3D 0, fname =3D 0x=
0, name =3D 0x0, <br>

=A0=A0=A0 filehand =3D 0x0}, parsed_uri =3D {scheme =3D 0x0, hostinfo =3D 0=
x0, user =3D 0x0, password =3D 0x0, hostname =3D 0x0, port_str =3D 0x0, pat=
h =3D 0x0, query =3D 0x0, fragment =3D 0x0, <br>=A0=A0=A0 hostent =3D 0x0, =
port =3D 0, is_initialized =3D 0, dns_looked_up =3D 0, dns_resolved =3D 0},=
 used_path_info =3D 0, per_dir_config =3D 0x0, request_config =3D 0x1eac9ba=
8, htaccess =3D 0x0, <br>

=A0 output_filters =3D 0x929aff8, input_filters =3D 0x1eaca3b0, proto_outpu=
t_filters =3D 0x929aff8, proto_input_filters =3D 0x1eaca3b0, eos_sent =3D 0=
}<br>(gdb) dump_table(r-&gt;headers_in)<br>[0] &#39;Server&#39;=3D&#39;Apac=
he-Coyote/1.1&#39; [0x1eaca1e0]<br>

[1] &#39;X-Powered-By&#39;=3D&#39;Servlet 2.4; <a href=3D"http://JBoss-4.2.=
2.GA" target=3D"_blank">JBoss-4.2.2.GA</a> (build: SVNTag=3DJBoss_4_2_2_GA =
date=3D200710221139)/Tomcat-5.5&#39; [0x1eaca218]<br>[2] &#39;Content-Type&=
#39;=3D&#39;text/xml;charset=3Dutf-8&#39; [0x1eaca290]<br>

[3] &#39;Transfer-Encoding&#39;=3D&#39;chunked&#39; [0x1eaca2d0]<br>[4] =
9;Date&#39;=3D&#39;Fri, 10 May 2013 11:05:59 GMT&#39; [0x1eaca310]<br><br><=
/div><div>A &#39;Server&#39; header in the headers_in field, isn&#39;t that=
 just nonsense ? Also note the NULLs in for protocol and proto_num which I =
don&#39;t think should be there.<br>

<br></div><div>I got the feeling something was really wrong with that reque=
st, long before mod_security got to see it. Unfortunately, I don&#39;t know=
 the circumstances of the request since I only have access to the coredump =
file and nothing else.<br>
<br>Any hint on how to proceed with this, e.g. what to inspect ? I don&#39;=
t think it&#39;s only per_dir_config screwed up here.<br><br>Cheers &amp; t=
hanks,<br>=A0 Thomas<br></div>
</div>

--001a11c34b4a9b958e04dd72b869--