Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (nike.apache.org: domain of yingqi.lu@intel.com designates
 143.182.124.21 as permitted sender)
From: "Lu, Yingqi" <yingqi.lu@intel.com>
To: "dev@httpd.apache.org" <dev@httpd.apache.org>
Subject: RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT
 support
Thread-Topic: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT
 support
Thread-Index: AQHPb3yLWHtG+H65FkK77Cd1nQ/1dptAOatwgAJEoSA=
Date: Fri, 16 May 2014 02:34:07 +0000
Message-ID: 
 <9ACD5B67AAC5594CB6268234CF29CF9AA3758A32@ORSMSX113.amr.corp.intel.com>
References: 
 <9ACD5B67AAC5594CB6268234CF29CF9AA37532AE@ORSMSX113.amr.corp.intel.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3757792@ORSMSX113.amr.corp.intel.com>
 <E8907DF1-72CE-439E-B3AA-131E3AB6D40B@jaguNET.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3757D88@ORSMSX113.amr.corp.intel.com>
In-Reply-To: 
 <9ACD5B67AAC5594CB6268234CF29CF9AA3757D88@ORSMSX113.amr.corp.intel.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Now, I just got the message I sent out yesterday, what a long delay!

Actually, this morning I sent out several messages checking for other feedb=
ack/comments/questions. I guess those messages are still in the air since I=
 have not received any copy of those mails.

All, please let me know your feedback/comments/questions. Thanks very much =
for your time reviewing the patches.

Thanks,
Yingqi=20

-----Original Message-----
From: Lu, Yingqi [mailto:yingqi.lu@intel.com]=20
Sent: Wednesday, May 14, 2014 9:00 AM
To: dev@httpd.apache.org
Subject: RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT=
 support

Hi Jim,

Thanks very much for your email.

In the SO_REUSEPORT patch, SO_REUSEPORT support is checked inside listen.c =
file. If the feature is not supported on the OS (for example, Linux kernel =
< 3.9), it will fall back to the original behavior.=20

In the bucket patch, there is no need to check the params. With single list=
en statement, it is just the default behavior.=20

Please let me know if this answers your question.

Thanks,
Yingqi

-----Original Message-----
From: Jim Jagielski [mailto:jim@jaguNET.com]
Sent: Wednesday, May 14, 2014 6:57 AM
To: dev@httpd.apache.org
Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT=
 support

This is very cool!

mod_status assumes that sysconf() exists, but do we need to do a config che=
ck on the params we use in these patches?
We look OK on Linux, FreeBSD and OSX...

I'm +1 on folding into trunk.

On May 13, 2014, at 7:55 PM, Lu, Yingqi <yingqi.lu@intel.com> wrote:

> Dear All,
> =20
> During the last couple weeks, I spent some time extending the original tw=
o patches from prefork MPM only to all three Linux MPMs (prefork, worker an=
d event). Attached is the latest version of the two patches. Bugzilla datab=
ase has also been updated already. The ID for the two patches are #55897 an=
d #56279. Please refer to messages below for details on both of the patches=
.
> =20
> Quick test result on modern dual socket Intel platform (Linux Kernel
> 3.13.9) SO_REUSEPORT patch (bugzilla #55897)
> 1.       Prefork MPM: 1 listen statement: 2.16X throughput improvement; 2=
 listen statements: 2.33X throughput improvement
> 2.       Worker MPM: 1 listen statement: 10% throughput improvement; 2 li=
sten statements: 35% throughput improvement
> 3.       Event MPM: 1 listen statement: 13% throughput improvement; 2 lis=
ten statements: throughput parity, but 62% response time reduction (with pa=
tch, 38% response time as original SW)
> =20
> Bucket patch (bugzilla #56279, only impact multiple listen statement case=
)
> 1.       Prefork MPM: 2 listen statements: 42% throughput improvement
> 2.       Worker MPM: 2 listen statements: 7% throughput improvement
> =20
> In all the above testing cases, significant response time reductions are =
observed, even with throughput improvements.
> =20
> Please let me know your feedback and comments.
> =20
> Thanks,
> Yingqi
> Software and workloads used in performance tests may have been optimized =
for performance only on Intel microprocessors. Performance tests, such as S=
YSmark and MobileMark, are measured using specific computer systems, compon=
ents, software, operations and functions. Any change to any of those factor=
s may cause the results to vary. You should consult other information and p=
erformance tests to assist you in fully evaluating your contemplated purcha=
ses, including the performance of that product when combined with other pro=
ducts.
> =20
> From: Lu, Yingqi [mailto:yingqi.lu@intel.com]
> Sent: Monday, March 17, 2014 1:41 PM
> To: dev@httpd.apache.org
> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
> SO_REUSEPORT support
> =20
> Dear all,
> =20
> Based on the feedback we received, we modified this patch. Here is the mo=
st recent version. We also modified the Bugzilla database(Bugzilla# 55897 f=
or SO_REUSEPORT patch; Bugzilla# 56279 for bucket patch).
> =20
> Below are the changes we made into this new version:
> =20
> According to Yann Ylavic and other people's comments, we separate the ori=
ginal patch between with and without SO_REUSEPORT into two separated patche=
s. The SO_REUSEPORT patch does not change the original listen sockets, it j=
ust duplicate the original one into multiple ones. Since the listen sockets=
 are identical, there is no need to change the idle_server_maintenance func=
tion. The bucket patch (without SO_REUSEPORT), on the other hand, it breaks=
 down the original listen record (if there are multiple listen socks) to mu=
ltiple listen record linked lists. In this case, idle_server_maintenance is=
 implemented at bucket level to address the situation that imbalanced traff=
ic occurs among different listen sockets/children buckets. In the bucket pa=
tch, the polling in the child process is removed since each child only list=
ens to 1 sock.
> =20
> According to Arkadiusz Miskiewicz's comment, we make the "detection of SO=
_REUSEPORT" at run time.
> =20
> According to Jeff Trawick's comments,
> 1. We generate the patches against the httpd trunk.
> 2. We tested the current patches and they do not impact event and worker =
mpms. If current patches can be accepted, we would be happy to extend them =
to other Linux based mpms. There are not much code changes, but require som=
e time to setup the workload to test.
> 3. We removed unnecessary comments and changed APLOGNO(). We also changed=
 some of the parameter/variable/function names to better represent their me=
anings.
> 4. There should be no build-in limitations for SO_REUSEPORT patch. For bu=
cket patch, the only thing is the number of children bucket only scales to =
MAX_SPAWN_RATE. If there are more than 32 (current default MAX_SPQN_RATE) l=
isten statements specified in the httpd.conf, the number of buckets will be=
 fixed to 32. The reason for this is because that we implement the idle_ser=
ver_maintenance at bucket level, each bucket's own max_spawn_rate is set to=
 MAX_SPAWN_RATE/num_buckets.
> =20
> Again, thanks very much for all the comments and feedback. Please let us =
know if there are more changes we need to complete to make them accepted.
> =20
> Thanks,
> Yingqi Lu
> =20
> =20
> =20
> From: Lu, Yingqi
> Sent: Tuesday, March 04, 2014 10:43 AM
> To: dev@httpd.apache.org
> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
> SO_REUSEPORT support
> =20
> Hi Jeff,
> =20
> Thanks very much for your time reviewing the patch! We will modify the pa=
tch according to your comments and repost it here.
> =20
> Thanks,
> Yingqi
> =20
> From: Jeff Trawick [mailto:trawick@gmail.com]
> Sent: Tuesday, March 04, 2014 10:08 AM
> To: Apache HTTP Server Development List
> Subject: Re: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
> SO_REUSEPORT support
> =20
> On Tue, Mar 4, 2014 at 10:35 AM, Lu, Yingqi <yingqi.lu@intel.com> wrote:
> Hi All,
> =20
> I just want to ping again on this patch to gather your feedback and comme=
nts. Please refer to the messages below for patch details.
> =20
> If you need any additional information/supporting data, please let us kno=
w as well.
> =20
> Yeah, it has been on my todo list, but I don't have time to give an in=20
> depth review at the moment.  Here are a few questions/comments.  (And=20
> you'll have to deal with the fact that it is unnecessarily tedious for=20
> me to evaluate higher-level considerations if there are a lot of=20
> distractions, such as the code comments below ;)  But others are of=20
> course free to chime in.)
> =20
> The patch should be against httpd trunk.  It probably won't take much tim=
e for you to create that patch and confirm basic operation.
> =20
> What is the impact to other MPMs, even if they shouldn't use or don't hav=
e the necessary code to use SO_REUSEPORT at this time?
> =20
> Have you tried the event MPM?
> =20
> Is there a way for the admin to choose this behavior?  Most won't care, b=
ut everyone's behavior is changed AFAICT.
> =20
> Are there built-in limitations in this patch that we should be aware of? =
 E.g., the free slot/spawn rate changes suggest to me that there can't be m=
ore than 1025 children???
> =20
> We should assume for now that there's no reason this couldn't be committe=
d to trunk after review/rework, so make sure it is as close as you can get =
it to what you think is the final form.
> =20
> For the configure-time check for 3.9 kernel: I think we'd also use=20
> AC_TRY_COMPILE at configure time to confirm that the SO_REUSEPORT=20
> definition is available, and not enable it if the system includes=20
> doesn't define it.  (Does that cause a problem for any significant=20
> number of people?)
> =20
> Don't mention the patch in the patch ;) (e.g., "This function is added=20
> for the patch.")
> =20
> Incomplete comments on style/syntax issues:
> =20
> * mixing declarations and statements (e.g., "duplr->next =3D 0;=20
> apr_socket_t *temps;") isn't supported by all compilers and is=20
> distracting when reviewing
> * suitable identifier names (e.g., fix global variable "flag" and=20
> whatever else isn't appropriate; "ap_post_config_listeners" should be=20
> renamed to indicate what it does
> * APLOGNO(99999) and comments about fixing it: Instead put "APLOGNO()"=20
> and don't add reminders in comments
> * this doesn't seem portable: "int free_slots[MAX_SPAWN_RATE/num_buckets]=
;"
> and so on
> =20
> =20
> Thanks,
> Yingqi
> =20
> =20
> From: Lu, Yingqi
> Sent: Friday, January 24, 2014 3:26 PM
> To: dev@httpd.apache.org
> Subject: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
> SO_REUSEPORT support
> =20
> Dear All,
> =20
> Our analysis of Apache httpd 2.4.7 prefork mpm, on 32 and 64 thread Intel=
 Xeon 2600 series systems, using an open source three tier social networkin=
g web server workload, revealed performance scaling issues.  In current sof=
tware single listen statement (listen 80) provides better scalability due t=
o un-serialized accept. However, when system is under very high load, this =
can lead to big number of child processes stuck in D state. On the other ha=
nd, the serialized accept approach cannot scale with the high load either. =
 In our analysis, a 32-thread system, with 2 listen statements specified, c=
ould scale to just 70% utilization, and a 64-thread system, with signal lis=
ten statement specified (listen 80, 4 network interfaces), could scale to o=
nly 60% utilization.
> =20
> Based on those findings, we created a prototype patch for prefork mpm whi=
ch extends performance and thread utilization. In Linux kernel newer than 3=
.9, SO_REUSEPORT is enabled. This feature allows multiple sockets listen to=
 the same IP:port and automatically round robins connections. We use this f=
eature to create multiple duplicated listener records of the original one a=
nd partition the child processes into buckets. Each bucket listens to 1 IP:=
port. In case of old kernel which does not have the SO_REUSEPORT enabled, w=
e modified the "multiple listen statement case" by creating 1 listen record=
 for each listen statement and partitioning the child processes into differ=
ent buckets. Each bucket listens to 1 IP:port.
> =20
> Quick tests of the patch, running the same workload, demonstrated a 22% t=
hroughput increase with 32-threads system and 2 listen statements (Linux ke=
rnel 3.10.4). With the older kernel (Linux Kernel 3.8.8, without SO_REUSEPO=
RT), 10% performance gain was measured. With single listen statement (liste=
n 80) configuration, we observed over 2X performance improvements on modern=
 dual socket Intel platforms (Linux Kernel 3.10.4). We also observed big re=
duction in response time, in addition to the throughput improvement gained =
in our tests 1.
> =20
> Following the feedback from the bugzilla website where we originally subm=
itted the patch, we removed the dependency of APR change to simplify the pa=
tch testing process. Thanks Jeff Trawick for his good suggestion! We are al=
so actively working on extending the patch to worker and event MPMs, as a n=
ext step. Meanwhile, we would like to gather comments from all of you on th=
e current prefork patch. Please take some time test it and let us know how =
it works in your environment.
> =20
> This is our first patch to the Apache community. Please help us review it=
 and let us know if there is anything we might revise to improve it. Your f=
eedback is very much appreciated.
> =20
> Configuration:
> <IfModule prefork.c>
>     ListenBacklog 105384
>     ServerLimit 105000
>     MaxClients 1024
>     MaxRequestsPerChild 0
>     StartServers 64
>     MinSpareServers 8
>     MaxSpareServers 16
> </IfModule>
> =20
> 1. Software and workloads used in performance tests may have been optimiz=
ed for performance only on Intel microprocessors. Performance tests, such a=
s SYSmark and MobileMark, are measured using specific computer systems, com=
ponents, software, operations and functions. Any change to any of those fac=
tors may cause the results to vary. You should consult other information an=
d performance tests to assist you in fully evaluating your contemplated pur=
chases, including the performance of that product when combined with other =
products.
> =20
> Thanks,
> Yingqi
>=20
>=20
> =20
> --
> Born in Roswell... married an alien...
> http://emptyhammock.com/
> http://edjective.org/
> =20
>=20
>=20
> =20
> --
> Born in Roswell... married an alien...
> http://emptyhammock.com/
> http://edjective.org/
> =20
> <httpd_trunk_so_reuseport.patch><httpd_trunk_bucket.patch>