From users-return-116906-archive-asf-public=cust-asf.ponee.io@httpd.apache.org Mon Feb 5 02:41:46 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 0669618064A for ; Mon, 5 Feb 2018 02:41:46 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id A7C9E160C59; Mon, 5 Feb 2018 01:41:45 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C89B3160C41 for ; Mon, 5 Feb 2018 02:41:44 +0100 (CET) Received: (qmail 68976 invoked by uid 500); 5 Feb 2018 01:41:38 -0000 Mailing-List: contact users-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: users@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@httpd.apache.org Received: (qmail 68964 invoked by uid 99); 5 Feb 2018 01:41:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Feb 2018 01:41:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B1A75C0335 for ; Mon, 5 Feb 2018 01:41:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.121 X-Spam-Level: X-Spam-Status: No, score=-0.121 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id yTQppC8n4iP0 for ; Mon, 5 Feb 2018 01:41:35 +0000 (UTC) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 72C275F1D5 for ; Mon, 5 Feb 2018 01:41:34 +0000 (UTC) Received: by mail-lf0-f51.google.com with SMTP id w27so39487351lfd.6 for ; Sun, 04 Feb 2018 17:41:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=kDTMcnUMQS/9Gz55nfkB73P0Pw+rJ1GEmhkt6fKTBBY=; b=K/xntKx9XXKeiWGamFa92kh2Lflm5f9DNWaTkYt4kKCXtIyYPGmeMCULdGnseTGqLA T+QnX+4X4XC20SE6qkQSjioFUYyTkKTBGGSY7s5Is7Bd1IAUoDZV5f7HQ8jez/WemJUw gba3Ul5pC3KI6bp/Ft6ZE/0a3HqroYgNgba7WEXvJKFktxKgHWuSHVYh98SJFj6R25ej tABCdrZfb65zIZ3ln1n+/iGoPPrti3LjG/iW3GPsg+LgbuMdgO+vHNOjEm2UqsNYq96j vEEj7JDpg9KChGY5hyk8Kp3VCu7WqnThWaYRsqzBMEyYLHTTmJH8PtXykzDEU6Y2XiVK XaLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=kDTMcnUMQS/9Gz55nfkB73P0Pw+rJ1GEmhkt6fKTBBY=; b=O9C7WGA4iwZ6OLIezeYd/Ye3JYORa8n7XmRmfcUIW06RHrEcpa/rlZU8xDFy5JW5b7 yVWtW01nYk5wIXRH2BfM7z8IU2gwG1t1rvKsrrc3lNVjY9Zn42ylv/iKxmpZqa5csgnQ E3NkHBKubHSzGKzAImUHNpadfGM7qgxv4IXZdASfUCrptYsZun9tt22JRn3MN9dvEqJX EHOYOGArjKQLFf4PRiIqR8JlP1hDF59FKZoVQFT+ZmEoBe+ZkRXvYx9VXJnUEGYjxBmJ jpRKx67DogK2d5KK49wxANflcYx+vXsezHFAqa43oli/yXbJKHx48YDZ8uJ/c/vkEHpN oqAg== X-Gm-Message-State: AKwxytcYUVEEJcJe5PTRoRf8mZTsRKANBC11CpkYvDprOCbwbK0OmfPY sTGB1gDUbQcyAFl/X5MTkbrio0WBpd7kXJz3EhAttQ== X-Google-Smtp-Source: AH8x224sfGNeCmdrpCZdV3ZRpZrI9/3EpERWZ6yrM+TCexRJEdaoBtgCiN+8BwNOTkUiLXdsq1xAO5Nz9VIylNBq9Uw= X-Received: by 10.25.44.147 with SMTP id s141mr26844831lfs.15.1517794893438; Sun, 04 Feb 2018 17:41:33 -0800 (PST) MIME-Version: 1.0 Received: by 10.46.1.166 with HTTP; Sun, 4 Feb 2018 17:41:32 -0800 (PST) In-Reply-To: References: <62731008-13a6-4784-05f9-065cc0373de6@gmx.de> <6e6f732a-6ec7-9a22-f497-49479feb37a9@gmx.de> <664e3aac-c2bb-5f1e-b632-36920e96aeea@gmx.de> From: Eric Covener Date: Sun, 4 Feb 2018 20:41:32 -0500 Message-ID: To: users@httpd.apache.org Content-Type: text/plain; charset="UTF-8" Subject: Re: [users@httpd] problems benchmarking php-fpm/proxy_fcgi with h2load On Sun, Feb 4, 2018 at 8:27 PM, Luca Toscano wrote: > Hi Hajo, > > > 2018-02-01 3:58 GMT+01:00 Luca Toscano : >> >> Hi Hajo, >> >> 2018-01-31 2:37 GMT-08:00 Hajo Locke : >>> >>> Hello, >>> >>> >>> Am 22.01.2018 um 11:54 schrieb Hajo Locke: >>> >>> Hello, >>> >>> Am 19.01.2018 um 15:48 schrieb Luca Toscano: >>> >>> Hi Hajo, >>> >>> 2018-01-19 13:23 GMT+01:00 Hajo Locke : >>>> >>>> Hello, >>>> >>>> thanks Daniel and Stefan. This is a good point. >>>> I did the test with a static file and this test was successfully done >>>> within only a few seconds. >>>> >>>> finished in 20.06s, 4984.80 req/s, 1.27GB/s >>>> requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 >>>> failed, 0 errored, 0 timeout >>>> >>>> so problem seems to be not h2load and basic apache. may be i should look >>>> deeper into proxy_fcgi configuration. >>>> php-fpm configuration is unchanged and was successfully used with >>>> classical fastcgi-benchmark, so i think i have to doublecheck the proxy. >>>> >>>> now i did this change in proxy: >>>> >>>> from >>>> enablereuse=on >>>> to >>>> enablereuse=off >>>> >>>> this change leads to a working h2load testrun: >>>> finished in 51.74s, 1932.87 req/s, 216.05MB/s >>>> requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 >>>> failed, 0 errored, 0 timeout >>>> >>>> iam surprised by that. i expected a higher performance when reusing >>>> backend connections rather then creating new ones. >>>> I did some further tests and changed some other php-fpm/proxy values, >>>> but once "enablereuse=on" is set, the problem returns. >>>> >>>> Should i just run the proxy with enablereuse=off? Or do you have an >>>> other suspicion? >>> >>> >>> >>> Before giving up I'd check two things: >>> >>> 1) That the same results happen with a regular localhost socket rather >>> than a unix one. >>> >>> I changed my setup to use tcp-sockets in php-fpm and proxy-fcgi. >>> Currently i see the same behaviour. >>> >>> 2) What changes on the php-fpm side. Are there more busy workers when >>> enablereuse is set to on? I am wondering how php-fpm handles FCGI requests >>> happening on the same socket, as opposed to assuming that 1 connection == 1 >>> FCGI request. >>> >>> If "enablereuse=off" is set i see a lot of running php-workerprocesses >>> (120-130) and high load. Behaviour is like expected. >>> When set "enablereuse=on" i can see a big change. number of running >>> php-workers is really low (~40). The test is running some time and then it >>> stucks. >>> I can see that php-fpm processes are still active and waiting for >>> connections, but proxy_fcgi is not using them nor it is establishing new >>> connections. loadavg is low and benchmarktest is not able to finalize. >>> >>> I did some further tests to solve this issue. I set ttl=1 for this Proxy >>> and achieved good performance and high number of working childs. But this is >>> paradoxical. >>> proxy_fcgi knows about inactive connection to kill it, but not reenable >>> this connection for working. >>> May be this is helpful to others. >>> >>> May be a kind of communicationproblem and checking health/busy status of >>> php-processes. >>> Whole proxy configuration is this: >>> >>> >>> ProxySet enablereuse=off flushpackets=On timeout=3600 max=15000 >>> >>> >>> SetHandler "proxy:fcgi://php70fpm" >>> >> >> >> Thanks a lot for following up and reporting these interesting results! >> Yann opened a thread[1] on dev@ to discuss the issue, let's follow up in >> there so we don't keep two conversations open. >> >> Luca >> >> [1]: >> https://lists.apache.org/thread.html/a9586dab96979bf45550c9714b36c49aa73526183998c5354ca9f1c8@%3Cdev.httpd.apache.org%3E >> > > reporting in here what I think it is happening in your test environment when > enablereuse is set to on. Recap of your settings: > > /etc/apache2/conf.d/limits.conf > StartServers 10 > MaxClients 500 > MinSpareThreads 450 > MaxSpareThreads 500 > ThreadsPerChild 150 > MaxRequestsPerChild 0 > Serverlimit 500 > > > ProxySet enablereuse=on flushpackets=On timeout=3600 max=1500 > > > SetHandler "proxy:fcgi://php70fpm/" > > > request_terminate_timeout = 7200 > listen = /dev/shm/php70fpm.sock > pm = ondemand > pm.max_children = 500 > pm.max_requests = 2000 > > By default mod_proxy allows a connection pool of ThreadsPerChild connections > to the backend for each httpd process, meanwhile in your case you have set > 3200 using the 'max' parameter (as stated in the docs it is a per process > setting, not a overall one). PHP-FPM handles one connection for each worker > at the time, and your settings allow a maximum of 500 processes, therefore a > maximum of 500 connections established at the same time from httpd. When > connection reuse is set to on, the side effect is that for each mod_proxy's > open/established connection in the pool there will be one PHP-FPM worker > tight to it, even if not serving any request (waiting for one basically). We should definitely spell out the impact of enablereuse on php-fpm specifically. I think one thing that is missing in the above recap is that w/o H2 given MaxClients=500, the impact is probably nil because there are simply not enough server threads to max out the pool and still try to make new connections. With H2, you have N additional h2 worker threads per process, so it puts it as out of balance as if MaxClients were larger but pm.max_children were not. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org For additional commands, e-mail: users-help@httpd.apache.org