From dev-return-90903-archive-asf-public=cust-asf.ponee.io@httpd.apache.org Sun Jan 21 13:42:35 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 09C4A180652 for ; Sun, 21 Jan 2018 13:42:35 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id EDAE5160C37; Sun, 21 Jan 2018 12:42:34 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 18891160C32 for ; Sun, 21 Jan 2018 13:42:33 +0100 (CET) Received: (qmail 21570 invoked by uid 500); 21 Jan 2018 12:42:32 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 21560 invoked by uid 99); 21 Jan 2018 12:42:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Jan 2018 12:42:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 679A71A0A4A for ; Sun, 21 Jan 2018 12:42:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.821 X-Spam-Level: X-Spam-Status: No, score=-0.821 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id SDOAU2Nw94a1 for ; Sun, 21 Jan 2018 12:42:30 +0000 (UTC) Received: from mail-qt0-f178.google.com (mail-qt0-f178.google.com [209.85.216.178]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 7CDF35F6CD for ; Sun, 21 Jan 2018 12:42:30 +0000 (UTC) Received: by mail-qt0-f178.google.com with SMTP id i1so14446362qtj.8 for ; Sun, 21 Jan 2018 04:42:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=ytb6nQowd8AERenYgW9Bl0CKCyedyEe6gEoT1Jz78Xs=; b=q3XCbgFR1bw/fQNY8mUxTPNCqwasVzIoUL3VgqaDY++SUD7NnR8JSyIFF6yLd8XsLn FZN61vKs0+WbbykakvHuCAVSSQOdOlZZPRyT001v3XQEnStBUmIaK9DxBfIVHlAAWZ7N 6d+lUTXCzOvlfgE0Zi1gfXbJ0LNuLytuMnSyuX8yyavdLWqwxMzo7mo5eCx0U/phIYSo pjbWBvKMvVCobhVFBSZsLtqYSzZZhmWTmhr5ahD18mrbRlP85nJn70kFvGVUcHTMGyAx AZgtggmp0u6O5foPNmOLqbst+WH2eWC5C7aZQST9206o2J+xsKT5MXBBbQ/MQLV3u5Qa sqpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=ytb6nQowd8AERenYgW9Bl0CKCyedyEe6gEoT1Jz78Xs=; b=kdRWDtnSdpN0l+e3oCvx2AmcU1Hhk/vI2GehVG0P7zNnXkUA+ZSzJzYQgUj+mBh3A5 GK3txkwGY7yAazEpIxRj6AjNmSEc3nizqHiQepWoERpcbZqGcykhSH2tiJQ4VWe1NDuh /PNNdRxgyLdzdLtXeJhj8IFrjFXYxXoiAu/oJzYW2cVDkmoaNBYEzrt2lvljmFsZbW6d uZH7n5IQdxuyAjF2WOD9oRtm/HRFwCnBvgD8snSiguCWfR6ENRwq3xxCx1oMXPQH6kPw H8mNCpFNSkqzubyPTT9b/RFcd8QtLRrvPaH0HkZABk7CGilYuBGkLI397TwQGpdrcyBq vbFQ== X-Gm-Message-State: AKwxytetIXuv6O6D91BNJ8Vhns/x5PVFF5XYbwPjS6yetlEo0mi2CdC2 VH12jznmjfJ48BSnZL5pnA8Ux/uevRVK+F0/BsCYEQ== X-Google-Smtp-Source: AH8x225FI/88y/vMQvZA3T64/UjDpkqmdfg5vmIm/yWm8pswlTayvYfTPzvlQTv7gRXJBWYgzJC+kFveoDGrD1jxrvI= X-Received: by 10.55.20.85 with SMTP id e82mr5969680qkh.100.1516538549904; Sun, 21 Jan 2018 04:42:29 -0800 (PST) MIME-Version: 1.0 Received: by 10.55.100.150 with HTTP; Sun, 21 Jan 2018 04:42:29 -0800 (PST) In-Reply-To: References: <62731008-13a6-4784-05f9-065cc0373de6@gmx.de> <8AC06961-32E5-4EAF-9849-5091D33F9409@greenbytes.de> From: Yann Ylavic Date: Sun, 21 Jan 2018 13:42:29 +0100 Message-ID: Subject: Re: [users@httpd] problems benchmarking php-fpm/proxy_fcgi with h2load To: httpd-dev Content-Type: text/plain; charset="UTF-8" [Moved from users@] On Sat, Jan 20, 2018 at 9:57 PM, Luca Toscano wrote: > > 2018-01-20 20:23 GMT+01:00 Luca Toscano : >> >> 2018-01-19 17:40 GMT+01:00 Yann Ylavic : >>> >>> On Fri, Jan 19, 2018 at 5:14 PM, Yann Ylavic >>> wrote: >>> > On Fri, Jan 19, 2018 at 1:46 PM, Daniel wrote: >>> >> I vaguely recall some issue with reuse when using unix socket files so >>> >> it was deliberately set to off by default, but yes, perhaps someone >>> >> experienced enough with mod_proxy_fcgi inner workings can shed some >>> >> light on this and the why yes/not. >>> >> >>> >> With socket files I never tried to enable "enablereuse=on" and got >>> >> much successful results, so perhaps it's safer to keep it off until >>> >> someone clarifies this issue, after all when dealing with unix sockets >>> >> the access delays are quite low. >>> > >>> > {en,dis}ablereuse has no effect on Unix Domain Sockets in mod_proxy, >>> > they are never reused. >>> >>> Well, actually it shouldn't, but while the code clearly doesn't reuse >>> sockets (creates a new one for each request), nothing seems to tell >>> the recycler that it should close them unconditionally at the end of >>> the request. >> >> >> Would you mind to point me to the snippet of code that does this? I am >> trying to reproduce the issue and see if there is a fd leak but didn't >> manage to so far.. Finally, after re-reading the code, I'm not sure the issue is in httpd. In ap_proxy_connect_backend(), we expect ap_proxy_check_connection() to check whether the connection can be reused or not, and it should work for Unix sockets too. I missed that call in my first analysis, so thought a new socket was always created... > > I am now able to reproduce with Hajo's settings, and indeed with > enablereuse=on I can see a lot of fds leaked via lsof: > > httpd 3230 3481 www-data 93u unix 0xffff9ada0cf60400 0t0 > 406770 type=STREAM > httpd 3230 3481 www-data 94u unix 0xffff9ada0cf60800 0t0 > 406773 type=STREAM > httpd 3230 3481 www-data 95u unix 0xffff9ada0cf66400 0t0 > 406776 type=STREAM > [..] Reusing connections means that we can have ThreadsPerChild sockets per backend (per child process). Is it really a leak (the more requests, the more sockets, never closed), or does it stop at ThreadsPerChild? > > With Yann's patch I cannot seem them anymore, anche h2load does not stop at > 50%/60% but completes without any issue. I am still not able to understand > why this happens reading the proxy_util.c code though :) My patch simply disables connections recycling/reuse for Unix sockets, but it's probably not the right since that's supposed to work. Could it be that php-fpm simply does not like connections keepalive/reuse? There may be a reason why we disablereuse by default in mod_proxy_fcgi... Where do communications stop working at 60%, something in the error log? Possibly some tcpdump/wireshark between proxy_fcgi and the backend could help here too. Regards, Yann.