httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Zahariev <fam...@famzah.net>
Subject Re: mod_fcgi: Excessive memory usage when large files are uploaded
Date Tue, 17 Jan 2017 15:49:57 GMT
<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Hi,</p>
    <p>Here is the patch --
<a class="moz-txt-link-freetext" href="https://github.com/famzah/mod_fcgid/commit/84c7c2dbf2047745c6aea87a4bc6b4061c98ac8e">https://github.com/famzah/mod_fcgid/commit/84c7c2dbf2047745c6aea87a4bc6b4061c98ac8e</a></p>
    <p>A few notes:</p>
    <ul>
      <li>I tested it with several files which had random data and
        various lengths: 0b, 127b, 128b, 129b, 150MB, 1MB, 1b, 250MB,
        350MB, 8191b, 8192b, 8193b. The uploaded size and SHA1 checksum
        match.<br>
      </li>
      <li>Also made sure that the special case "There are something
        left" in the code was tested too, at the end when "nvec != 0".</li>
      <li>I can't test this on Windows but porting the patch in
        "fcgid_proc_win.c" looks easy, since the same loop method is
        used there too. Additionally, "fcgid_proc_win.c" doesn't group
        write requests in 8 chunks and does everything one-by-one (one
        read, then one write to the IPC socket).</li>
    </ul>
    <br>
    Please review the patch. If it looks good, I'll test it on our
    production machines and then submit it for mainstream merge.<br>
    <br>
    Best regards.<br>
    --Ivan<br>
    <br>
    <div class="moz-cite-prefix">On 17.1.2017 г. 13:03 ч., Yann Ylavic
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAKQ1sVMrMpOwBAPZEBBroyqc3+KvQBjV99gyu_0N+sgjTR4JXA@mail.gmail.com"
      type="cite">
      <pre wrap="">On Tue, Jan 17, 2017 at 9:06 AM, Ivan Zahariev <a class="moz-txt-link-rfc2396E"
href="mailto:famzah@famzah.net">&lt;famzah@famzah.net&gt;</a> wrote:
</pre>
      <blockquote type="cite">
        <pre wrap="">
1. Delete each bucket after sending it to the "ipc_handle". I've looked through
the call tree and the *output_brigade is last used by proc_write_ipc().
Therefore, it should be safe to empty it while being processed there.
2. Take the same approach as mod_http2, which handles FILE buckets in a
different way. Instead of using apr_bucket_read(), they process FILE buckets
by apr_file_read() and manage the data buffer manually. This way the
original *output_brigade won't be modified and automatically split by
apr_bucket_read(). This requires more coding work.
</pre>
      </blockquote>
      <pre wrap="">
I would go for 1., but keep in mind that you can't delete the bucket
while it is still pointed to by the iovec...

An (optimal) alternative would be to use apr_socket_sendfile() (for
file buckets) like sendfile_nonblocking() does in httpd's core output
filter (see server/core_filters.c).
But 1. is wise still for non file buckets.

Regards,
Yann.
</pre>
    </blockquote>
    <br>
  </body>
</html>

Mime
View raw message