Return-Path: X-Original-To: apmail-httpd-dev-archive@www.apache.org Delivered-To: apmail-httpd-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 27B2810543 for ; Fri, 30 May 2014 14:10:14 +0000 (UTC) Received: (qmail 32125 invoked by uid 500); 30 May 2014 14:10:13 -0000 Delivered-To: apmail-httpd-dev-archive@httpd.apache.org Received: (qmail 32041 invoked by uid 500); 30 May 2014 14:10:13 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 32033 invoked by uid 99); 30 May 2014 14:10:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 May 2014 14:10:13 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS,TVD_FW_GRAPHIC_NAME_MID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of miacob@ca.ibm.com designates 32.97.182.138 as permitted sender) Received: from [32.97.182.138] (HELO e8.ny.us.ibm.com) (32.97.182.138) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 May 2014 14:10:08 +0000 Received: from /spool/local by e8.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 30 May 2014 10:09:47 -0400 Received: from d01dlp01.pok.ibm.com (9.56.250.166) by e8.ny.us.ibm.com (192.168.1.108) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 30 May 2014 10:09:44 -0400 Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id 5AF1B38C8027 for ; Fri, 30 May 2014 10:09:44 -0400 (EDT) Received: from d01av05.pok.ibm.com (d01av05.pok.ibm.com [9.56.224.195]) by b01cxnp23032.gho.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id s4UE9aqY9896388 for ; Fri, 30 May 2014 14:09:44 GMT Received: from d01av05.pok.ibm.com (localhost [127.0.0.1]) by d01av05.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s4UE9BHD009037 for ; Fri, 30 May 2014 10:09:11 -0400 Received: from d25ml01.torolab.ibm.com (d25ml01.torolab.ibm.com [9.26.29.94]) by d01av05.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id s4UE99m8008336 for ; Fri, 30 May 2014 10:09:09 -0400 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA37532AE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3757792@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3757D88@ORSMSX113.amr.corp.intel.com> <3AB38FA5-CA8D-4818-8DA7-8198AE01A8BB@jaguNET.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3758EA6@ORSMSX113.amr.corp.intel.com> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support X-KeepSent: 9C5E7A6E:9008C525-85257CE8:004DAFA8; type=4; name=$KeepSent To: dev@httpd.apache.org X-Mailer: IBM Notes Release 9.0 SHF141 June 05, 2013 Message-ID: From: Mihai Iacob Date: Fri, 30 May 2014 10:08:50 -0400 X-MIMETrack: Serialize by Router on D25ML01/25/M/IBM(Release 9.0.1|October 14, 2013) at 05/30/2014 10:09:09 MIME-Version: 1.0 Content-type: multipart/related; Boundary="0__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938" X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14053014-0320-0000-0000-0000036A89B0 X-Virus-Checked: Checked by ClamAV on apache.org --0__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938 Content-type: multipart/alternative; Boundary="1__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938" --1__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: quoted-printable how do I unsubsribe from this list ? Regards, Mihai Iacob DB2 Security Development DB2 pureScale Development Phone: (905) 413-5378 Email: miacob@ca.ibm.com From: Jim Jagielski To: dev@httpd.apache.org Date: 05/30/2014 10:07 AM Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support Thx! Let me review. My plan is to fold into trunk this weekend. On May 16, 2014, at 2:53 PM, Lu, Yingqi wrote: > Hi Jim, > > Thanks very much for clarifying this with me. I added #ifdef in the c= ode to check _SC_NPROCESSORS_ONLN in the so_reuseport patch. Bucket patch d= oes not use this parameter so that it remains the same. > > Attached are the two most recent patches. I already updated the bugzi= lla #55897 as well. > > Thanks, > Yingqi > > -----Original Message----- > From: Jim Jagielski [mailto:jim@jaguNET.com] > Sent: Thursday, May 15, 2014 7:53 AM > To: dev@httpd.apache.org > Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support > > I was thinking more about the sysconf(_SC_NPROCESSORS_ONLN) stuff... > We could either check for that during config/build or protect it with= a #ifdef in the code (and create some logging so the admin nows if it was= found or not). > > On May 14, 2014, at 11:59 AM, Lu, Yingqi wrote:= > >> Hi Jim, >> >> Thanks very much for your email. >> >> In the SO_REUSEPORT patch, SO_REUSEPORT support is checked inside listen.c file. If the feature is not supported on the OS (for example, Linux kernel < 3.9), it will fall back to the original behavior. >> >> In the bucket patch, there is no need to check the params. With sing= le listen statement, it is just the default behavior. >> >> Please let me know if this answers your question. >> >> Thanks, >> Yingqi >> >> -----Original Message----- >> From: Jim Jagielski [mailto:jim@jaguNET.com] >> Sent: Wednesday, May 14, 2014 6:57 AM >> To: dev@httpd.apache.org >> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with >> SO_REUSEPORT support >> >> This is very cool! >> >> mod_status assumes that sysconf() exists, but do we need to do a con= fig check on the params we use in these patches? >> We look OK on Linux, FreeBSD and OSX... >> >> I'm +1 on folding into trunk. >> >> On May 13, 2014, at 7:55 PM, Lu, Yingqi wrote:= >> >>> Dear All, >>> >>> During the last couple weeks, I spent some time extending the origi= nal two patches from prefork MPM only to all three Linux MPMs (prefork, wor= ker and event). Attached is the latest version of the two patches. Bugzilla= database has also been updated already. The ID for the two patches are #55897 and #56279. Please refer to messages below for details on both o= f the patches. >>> >>> Quick test result on modern dual socket Intel platform (Linux Kerne= l >>> 3.13.9) SO_REUSEPORT patch (bugzilla #55897) >>> 1. Prefork MPM: 1 listen statement: 2.16X throughput improvem= ent; 2 listen statements: 2.33X throughput improvement >>> 2. Worker MPM: 1 listen statement: 10% throughput improvement= ; 2 listen statements: 35% throughput improvement >>> 3. Event MPM: 1 listen statement: 13% throughput improvement;= 2 listen statements: throughput parity, but 62% response time reduction (= with patch, 38% response time as original SW) >>> >>> Bucket patch (bugzilla #56279, only impact multiple listen statemen= t case) >>> 1. Prefork MPM: 2 listen statements: 42% throughput improveme= nt >>> 2. Worker MPM: 2 listen statements: 7% throughput improvement= >>> >>> In all the above testing cases, significant response time reduction= s are observed, even with throughput improvements. >>> >>> Please let me know your feedback and comments. >>> >>> Thanks, >>> Yingqi >>> Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance te= sts, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to = any of those factors may cause the results to vary. You should consult othe= r information and performance tests to assist you in fully evaluating you= r contemplated purchases, including the performance of that product when combined with other products. >>> >>> From: Lu, Yingqi [mailto:yingqi.lu@intel.com] >>> Sent: Monday, March 17, 2014 1:41 PM >>> To: dev@httpd.apache.org >>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch with >>> SO_REUSEPORT support >>> >>> Dear all, >>> >>> Based on the feedback we received, we modified this patch. Here is = the most recent version. We also modified the Bugzilla database(Bugzilla# 5= 5897 for SO_REUSEPORT patch; Bugzilla# 56279 for bucket patch). >>> >>> Below are the changes we made into this new version: >>> >>> According to Yann Ylavic and other people's comments, we separate t= he original patch between with and without SO_REUSEPORT into two separated= patches. The SO_REUSEPORT patch does not change the original listen sockets, it just duplicate the original one into multiple ones. Since t= he listen sockets are identical, there is no need to change the idle_server_maintenance function. The bucket patch (without SO_REUSEPOR= T), on the other hand, it breaks down the original listen record (if there = are multiple listen socks) to multiple listen record linked lists. In this case, idle_server_maintenance is implemented at bucket level to address= the situation that imbalanced traffic occurs among different listen sockets/children buckets. In the bucket patch, the polling in the child= process is removed since each child only listens to 1 sock. >>> >>> According to Arkadiusz Miskiewicz's comment, we make the "detection= of SO_REUSEPORT" at run time. >>> >>> According to Jeff Trawick's comments, 1. We generate the patches >>> against the httpd trunk. >>> 2. We tested the current patches and they do not impact event and worker mpms. If current patches can be accepted, we would be happy to extend them to other Linux based mpms. There are not much code changes,= but require some time to setup the workload to test. >>> 3. We removed unnecessary comments and changed APLOGNO(). We also changed some of the parameter/variable/function names to better represe= nt their meanings. >>> 4. There should be no build-in limitations for SO_REUSEPORT patch. = For bucket patch, the only thing is the number of children bucket only scal= es to MAX_SPAWN_RATE. If there are more than 32 (current default MAX_SPQN_RATE) listen statements specified in the httpd.conf, the numbe= r of buckets will be fixed to 32. The reason for this is because that we implement the idle_server_maintenance at bucket level, each bucket's ow= n max_spawn_rate is set to MAX_SPAWN_RATE/num_buckets. >>> >>> Again, thanks very much for all the comments and feedback. Please l= et us know if there are more changes we need to complete to make them accepted. >>> >>> Thanks, >>> Yingqi Lu >>> >>> >>> >>> From: Lu, Yingqi >>> Sent: Tuesday, March 04, 2014 10:43 AM >>> To: dev@httpd.apache.org >>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch with >>> SO_REUSEPORT support >>> >>> Hi Jeff, >>> >>> Thanks very much for your time reviewing the patch! We will modify = the patch according to your comments and repost it here. >>> >>> Thanks, >>> Yingqi >>> >>> From: Jeff Trawick [mailto:trawick@gmail.com] >>> Sent: Tuesday, March 04, 2014 10:08 AM >>> To: Apache HTTP Server Development List >>> Subject: Re: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch with >>> SO_REUSEPORT support >>> >>> On Tue, Mar 4, 2014 at 10:35 AM, Lu, Yingqi wrote: >>> Hi All, >>> >>> I just want to ping again on this patch to gather your feedback and= comments. Please refer to the messages below for patch details. >>> >>> If you need any additional information/supporting data, please let = us know as well. >>> >>> Yeah, it has been on my todo list, but I don't have time to give an= >>> in depth review at the moment. Here are a few questions/comments. >>> (And you'll have to deal with the fact that it is unnecessarily >>> tedious for me to evaluate higher-level considerations if there are= a >>> lot of distractions, such as the code comments below ;) But others= >>> are of course free to chime in.) >>> >>> The patch should be against httpd trunk. It probably won't take mu= ch time for you to create that patch and confirm basic operation. >>> >>> What is the impact to other MPMs, even if they shouldn't use or don= 't have the necessary code to use SO_REUSEPORT at this time? >>> >>> Have you tried the event MPM? >>> >>> Is there a way for the admin to choose this behavior? Most won't c= are, but everyone's behavior is changed AFAICT. >>> >>> Are there built-in limitations in this patch that we should be awar= e of? E.g., the free slot/spawn rate changes suggest to me that there ca= n't be more than 1025 children??? >>> >>> We should assume for now that there's no reason this couldn't be committed to trunk after review/rework, so make sure it is as close as = you can get it to what you think is the final form. >>> >>> For the configure-time check for 3.9 kernel: I think we'd also use >>> AC_TRY_COMPILE at configure time to confirm that the SO_REUSEPORT >>> definition is available, and not enable it if the system includes >>> doesn't define it. (Does that cause a problem for any significant >>> number of people?) >>> >>> Don't mention the patch in the patch ;) (e.g., "This function is >>> added for the patch.") >>> >>> Incomplete comments on style/syntax issues: >>> >>> * mixing declarations and statements (e.g., "duplr->next =3D 0; >>> apr_socket_t *temps;") isn't supported by all compilers and is >>> distracting when reviewing >>> * suitable identifier names (e.g., fix global variable "flag" and >>> whatever else isn't appropriate; "ap_post_config_listeners" should = be >>> renamed to indicate what it does >>> * APLOGNO(99999) and comments about fixing it: Instead put "APLOGNO= ()" >>> and don't add reminders in comments >>> * this doesn't seem portable: "int free_slots [MAX_SPAWN_RATE/num_buckets];" >>> and so on >>> >>> >>> Thanks, >>> Yingqi >>> >>> >>> From: Lu, Yingqi >>> Sent: Friday, January 24, 2014 3:26 PM >>> To: dev@httpd.apache.org >>> Subject: [PATCH ASF bugzilla# 55897]prefork_mpm patch with >>> SO_REUSEPORT support >>> >>> Dear All, >>> >>> Our analysis of Apache httpd 2.4.7 prefork mpm, on 32 and 64 thread= Intel Xeon 2600 series systems, using an open source three tier social networking web server workload, revealed performance scaling issues. I= n current software single listen statement (listen 80) provides better scalability due to un-serialized accept. However, when system is under = very high load, this can lead to big number of child processes stuck in D st= ate. On the other hand, the serialized accept approach cannot scale with the= high load either. In our analysis, a 32-thread system, with 2 listen statements specified, could scale to just 70% utilization, and a 64-thr= ead system, with signal listen statement specified (listen 80, 4 network interfaces), could scale to only 60% utilization. >>> >>> Based on those findings, we created a prototype patch for prefork m= pm which extends performance and thread utilization. In Linux kernel newer= than 3.9, SO_REUSEPORT is enabled. This feature allows multiple sockets= listen to the same IP:port and automatically round robins connections. = We use this feature to create multiple duplicated listener records of the original one and partition the child processes into buckets. Each bucke= t listens to 1 IP:port. In case of old kernel which does not have the SO_REUSEPORT enabled, we modified the "multiple listen statement case" = by creating 1 listen record for each listen statement and partitioning the= child processes into different buckets. Each bucket listens to 1 IP:por= t. >>> >>> Quick tests of the patch, running the same workload, demonstrated a= 22% throughput increase with 32-threads system and 2 listen statements (Lin= ux kernel 3.10.4). With the older kernel (Linux Kernel 3.8.8, without SO_REUSEPORT), 10% performance gain was measured. With single listen statement (listen 80) configuration, we observed over 2X performance improvements on modern dual socket Intel platforms (Linux Kernel 3.10.4= ). We also observed big reduction in response time, in addition to the throughput improvement gained in our tests 1. >>> >>> Following the feedback from the bugzilla website where we originall= y submitted the patch, we removed the dependency of APR change to simplif= y the patch testing process. Thanks Jeff Trawick for his good suggestion!= We are also actively working on extending the patch to worker and event MP= Ms, as a next step. Meanwhile, we would like to gather comments from all of= you on the current prefork patch. Please take some time test it and let us = know how it works in your environment. >>> >>> This is our first patch to the Apache community. Please help us rev= iew it and let us know if there is anything we might revise to improve it. = Your feedback is very much appreciated. >>> >>> Configuration: >>> >>> ListenBacklog 105384 >>> ServerLimit 105000 >>> MaxClients 1024 >>> MaxRequestsPerChild 0 >>> StartServers 64 >>> MinSpareServers 8 >>> MaxSpareServers 16 >>> >>> >>> 1. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance te= sts, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to = any of those factors may cause the results to vary. You should consult othe= r information and performance tests to assist you in fully evaluating you= r contemplated purchases, including the performance of that product when combined with other products. >>> >>> Thanks, >>> Yingqi >>> >>> >>> >>> -- >>> Born in Roswell... married an alien... >>> http://emptyhammock.com/ >>> http://edjective.org/ >>> >>> >>> >>> >>> -- >>> Born in Roswell... married an alien... >>> http://emptyhammock.com/ >>> http://edjective.org/ >>> >>> >> > > = --1__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938 Content-type: text/html; charset=US-ASCII Content-Disposition: inline Content-transfer-encoding: quoted-printable

how do I unsubsribe from this l= ist ?

Regards,
Mihai Iacob
DB2 Security Development
DB2 pureScale Development
Phone: (905) 413-5378
Email: miacob@ca.ibm.com

3D"InactiveJim Jagielski ---05/30/2014 10:07:53 AM---Thx! Let me review= . My plan is to fold into trunk this weekend.

From: Jim Jagielski <jim@jaguNET.com>=
To: dev@httpd.apache.org
Date: 05/30/2014 10:07 AM
Subject: = Re: [PATCH ASF bugzilla# 55897]pre= fork_mpm patch with SO_REUSEPORT support




Thx! Let me review. My plan is to fold into trunk<= br> this weekend.

On May 16, 2014, at 2:53 PM, Lu, Yingqi <yingqi.lu@intel.com> wro= te:

> Hi Jim,
>
> Thanks very much for clarifying this with me. I added #ifdef in th= e code to check _SC_NPROCESSORS_ONLN in the so_reuseport patch. Bucket = patch does not use this parameter so that it remains the same.
>
> Attached are the two most recent patches. I already updated the bu= gzilla #55897 as well.
>
> Thanks,
> Yingqi
>
> -----Original Message-----
> From: Jim Jagielski [mailto:jim@jaguNET.com]
> Sent: Thursday, May 15, 2014 7:53 AM
> To: dev@httpd.apache.org
> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_= REUSEPORT support
>
> I was thinking more about the sysconf(_SC_NPROCESSORS_ONLN) stuff.= ..
> We could either check for that during config/build or protect it w= ith a #ifdef in the code (and create some logging so the admin nows if = it was found or not).
>
> On May 14, 2014, at 11:59 AM, Lu, Yingqi <yingqi.lu@intel.com&g= t; wrote:
>
>> Hi Jim,
>>
>> Thanks very much for your email.
>>
>> In the SO_REUSEPORT patch, SO_REUSEPORT support is checked ins= ide listen.c file. If the feature is not supported on the OS (for examp= le, Linux kernel < 3.9), it will fall back to the original behavior.=
>>
>> In the bucket patch, there is no need to check the params. Wit= h single listen statement, it is just the default behavior.
>>
>> Please let me know if this answers your question.
>>
>> Thanks,
>> Yingqi
>>
>> -----Original Message-----
>> From: Jim Jagielski [mailto:jim@jaguNET.com]
>> Sent: Wednesday, May 14, 2014 6:57 AM
>> To: dev@httpd.apache.org
>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=
>> SO_REUSEPORT support
>>
>> This is very cool!
>>
>> mod_status assumes that sysconf() exists, but do we need to do= a config check on the params we use in these patches?
>> We look OK on Linux, FreeBSD and OSX...
>>
>> I'm +1 on folding into trunk.
>>
>> On May 13, 2014, at 7:55 PM, Lu, Yingqi <yingqi.lu@intel.co= m> wrote:
>>
>>> Dear All,
>>>
>>> During the last couple weeks, I spent some time extending = the original two patches from prefork MPM only to all three Linux MPMs = (prefork, worker and event). Attached is the latest version of the two = patches. Bugzilla database has also been updated already. The ID for th= e two patches are #55897 and #56279. Please refer to messages below for= details on both of the patches.
>>>
>>> Quick test result on modern dual socket Intel platform (Li= nux Kernel
>>> 3.13.9) SO_REUSEPORT patch (bugzilla #55897)
>>> 1.       Prefork MPM: 1 listen statement: 2= .16X throughput improvement; 2 listen statements: 2.33X throughput impr= ovement
>>> 2.       Worker MPM: 1 listen statement: 10= % throughput improvement; 2 listen statements: 35% throughput improveme= nt
>>> 3.       Event MPM: 1 listen statement: 13%= throughput improvement; 2 listen statements: throughput parity, but 62= % response time reduction (with patch, 38% response time as original SW= )
>>>
>>> Bucket patch (bugzilla #56279, only impact multiple listen= statement case)
>>> 1.       Prefork MPM: 2 listen statements: = 42% throughput improvement
>>> 2.       Worker MPM: 2 listen statements: 7= % throughput improvement
>>>
>>> In all the above testing cases, significant response time = reductions are observed, even with throughput improvements.
>>>
>>> Please let me know your feedback and comments.
>>>
>>> Thanks,
>>> Yingqi
>>> Software and workloads used in performance tests may have = been optimized for performance only on Intel microprocessors. Performan= ce tests, such as SYSmark and MobileMark, are measured using specific c= omputer systems, components, software, operations and functions. Any ch= ange to any of those factors may cause the results to vary. You should = consult other information and performance tests to assist you in fully = evaluating your contemplated purchases, including the performance of th= at product when combined with other products.
>>>
>>> From: Lu, Yingqi [mailto:yingqi.lu@intel.com]
>>> Sent: Monday, March 17, 2014 1:41 PM
>>> To: dev@httpd.apache.org
>>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm pa= tch with
>>> SO_REUSEPORT support
>>>
>>> Dear all,
>>>
>>> Based on the feedback we received, we modified this patch.= Here is the most recent version. We also modified the Bugzilla databas= e(Bugzilla# 55897 for SO_REUSEPORT patch; Bugzilla# 56279 for bucket pa= tch).
>>>
>>> Below are the changes we made into this new version:
>>>
>>> According to Yann Ylavic and other people's comments, we s= eparate the original patch between with and without SO_REUSEPORT into t= wo separated patches. The SO_REUSEPORT patch does not change the origin= al listen sockets, it just duplicate the original one into multiple one= s. Since the listen sockets are identical, there is no need to change t= he idle_server_maintenance function. The bucket patch (without SO_REUSE= PORT), on the other hand, it breaks down the original listen record (if= there are multiple listen socks) to multiple listen record linked list= s. In this case, idle_server_maintenance is implemented at bucket level= to address the situation that imbalanced traffic occurs among differen= t listen sockets/children buckets. In the bucket patch, the polling in = the child process is removed since each child only listens to 1 sock. >>>
>>> According to Arkadiusz Miskiewicz's comment, we make the &= quot;detection of SO_REUSEPORT" at run time.
>>>
>>> According to Jeff Trawick's comments, 1. We generate the p= atches
>>> against the httpd trunk.
>>> 2. We tested the current patches and they do not impact ev= ent and worker mpms. If current patches can be accepted, we would be ha= ppy to extend them to other Linux based mpms. There are not much code c= hanges, but require some time to setup the workload to test.
>>> 3. We removed unnecessary comments and changed APLOGNO(). = We also changed some of the parameter/variable/function names to better= represent their meanings.
>>> 4. There should be no build-in limitations for SO_REUSEPOR= T patch. For bucket patch, the only thing is the number of children buc= ket only scales to MAX_SPAWN_RATE. If there are more than 32 (current d= efault MAX_SPQN_RATE) listen statements specified in the httpd.conf, th= e number of buckets will be fixed to 32. The reason for this is because= that we implement the idle_server_maintenance at bucket level, each bu= cket's own max_spawn_rate is set to MAX_SPAWN_RATE/num_buckets.
>>>
>>> Again, thanks very much for all the comments and feedback.= Please let us know if there are more changes we need to complete to ma= ke them accepted.
>>>
>>> Thanks,
>>> Yingqi Lu
>>>
>>>
>>>
>>> From: Lu, Yingqi
>>> Sent: Tuesday, March 04, 2014 10:43 AM
>>> To: dev@httpd.apache.org
>>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm pa= tch with
>>> SO_REUSEPORT support
>>>
>>> Hi Jeff,
>>>
>>> Thanks very much for your time reviewing the patch! We wil= l modify the patch according to your comments and repost it here.
>>>
>>> Thanks,
>>> Yingqi
>>>
>>> From: Jeff Trawick [mailto:trawick@gmail.com]
>>> Sent: Tuesday, March 04, 2014 10:08 AM
>>> To: Apache HTTP Server Development List
>>> Subject: Re: FW: [PATCH ASF bugzilla# 55897]prefork_mpm pa= tch with
>>> SO_REUSEPORT support
>>>
>>> On Tue, Mar 4, 2014 at 10:35 AM, Lu, Yingqi <yingqi.lu@= intel.com> wrote:
>>> Hi All,
>>>
>>> I just want to ping again on this patch to gather your fee= dback and comments. Please refer to the messages below for patch detail= s.
>>>
>>> If you need any additional information/supporting data, pl= ease let us know as well.
>>>
>>> Yeah, it has been on my todo list, but I don't have time t= o give an
>>> in depth review at the moment.  Here are a few questi= ons/comments.  
>>> (And you'll have to deal with the fact that it is unnecess= arily
>>> tedious for me to evaluate higher-level considerations if = there are a
>>> lot of distractions, such as the code comments below ;) &n= bsp;But others
>>> are of course free to chime in.)
>>>
>>> The patch should be against httpd trunk.  It probably= won't take much time for you to create that patch and confirm basic op= eration.
>>>
>>> What is the impact to other MPMs, even if they shouldn't u= se or don't have the necessary code to use SO_REUSEPORT at this time? >>>
>>> Have you tried the event MPM?
>>>
>>> Is there a way for the admin to choose this behavior? &nbs= p;Most won't care, but everyone's behavior is changed AFAICT.
>>>
>>> Are there built-in limitations in this patch that we shoul= d be aware of?  E.g., the free slot/spawn rate changes suggest to = me that there can't be more than 1025 children???
>>>
>>> We should assume for now that there's no reason this could= n't be committed to trunk after review/rework, so make sure it is as cl= ose as you can get it to what you think is the final form.
>>>
>>> For the configure-time check for 3.9 kernel: I think we'd = also use
>>> AC_TRY_COMPILE at configure time to confirm that the SO_RE= USEPORT
>>> definition is available, and not enable it if the system i= ncludes
>>> doesn't define it.  (Does that cause a problem for an= y significant
>>> number of people?)
>>>
>>> Don't mention the patch in the patch ;) (e.g., "This = function is
>>> added for the patch.")
>>>
>>> Incomplete comments on style/syntax issues:
>>>
>>> * mixing declarations and statements (e.g., "duplr-&g= t;next =3D 0;
>>> apr_socket_t *temps;") isn't supported by all compile= rs and is
>>> distracting when reviewing
>>> * suitable identifier names (e.g., fix global variable &qu= ot;flag" and
>>> whatever else isn't appropriate; "ap_post_config_list= eners" should be
>>> renamed to indicate what it does
>>> * APLOGNO(99999) and comments about fixing it: Instead put= "APLOGNO()"
>>> and don't add reminders in comments
>>> * this doesn't seem portable: "int free_slots[MAX_SPA= WN_RATE/num_buckets];"
>>> and so on
>>>
>>>
>>> Thanks,
>>> Yingqi
>>>
>>>
>>> From: Lu, Yingqi
>>> Sent: Friday, January 24, 2014 3:26 PM
>>> To: dev@httpd.apache.org
>>> Subject: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=
>>> SO_REUSEPORT support
>>>
>>> Dear All,
>>>
>>> Our analysis of Apache httpd 2.4.7 prefork mpm, on 32 and = 64 thread Intel Xeon 2600 series systems, using an open source three ti= er social networking web server workload, revealed performance scaling = issues.  In current software single listen statement (listen 80) p= rovides better scalability due to un-serialized accept. However, when s= ystem is under very high load, this can lead to big number of child pro= cesses stuck in D state. On the other hand, the serialized accept appro= ach cannot scale with the high load either.  In our analysis, a 32= -thread system, with 2 listen statements specified, could scale to just= 70% utilization, and a 64-thread system, with signal listen statement = specified (listen 80, 4 network interfaces), could scale to only 60% ut= ilization.
>>>
>>> Based on those findings, we created a prototype patch for = prefork mpm which extends performance and thread utilization. In Linux = kernel newer than 3.9, SO_REUSEPORT is enabled. This feature allows mul= tiple sockets listen to the same IP:port and automatically round robins= connections. We use this feature to create multiple duplicated listene= r records of the original one and partition the child processes into bu= ckets. Each bucket listens to 1 IP:port. In case of old kernel which do= es not have the SO_REUSEPORT enabled, we modified the "multiple li= sten statement case" by creating 1 listen record for each listen s= tatement and partitioning the child processes into different buckets. E= ach bucket listens to 1 IP:port.
>>>
>>> Quick tests of the patch, running the same workload, demon= strated a 22% throughput increase with 32-threads system and 2 listen s= tatements (Linux kernel 3.10.4). With the older kernel (Linux Kernel 3.= 8.8, without SO_REUSEPORT), 10% performance gain was measured. With sin= gle listen statement (listen 80) configuration, we observed over 2X per= formance improvements on modern dual socket Intel platforms (Linux Kern= el 3.10.4). We also observed big reduction in response time, in additio= n to the throughput improvement gained in our tests 1.
>>>
>>> Following the feedback from the bugzilla website where we = originally submitted the patch, we removed the dependency of APR change= to simplify the patch testing process. Thanks Jeff Trawick for his goo= d suggestion! We are also actively working on extending the patch to wo= rker and event MPMs, as a next step. Meanwhile, we would like to gather= comments from all of you on the current prefork patch. Please take som= e time test it and let us know how it works in your environment.
>>>
>>> This is our first patch to the Apache community. Please he= lp us review it and let us know if there is anything we might revise to= improve it. Your feedback is very much appreciated.
>>>
>>> Configuration:
>>> <IfModule prefork.c>
>>>   ListenBacklog 105384
>>>   ServerLimit 105000
>>>   MaxClients 1024
>>>   MaxRequestsPerChild 0
>>>   StartServers 64
>>>   MinSpareServers 8
>>>   MaxSpareServers 16
>>> </IfModule>
>>>
>>> 1. Software and workloads used in performance tests may ha= ve been optimized for performance only on Intel microprocessors. Perfor= mance tests, such as SYSmark and MobileMark, are measured using specifi= c computer systems, components, software, operations and functions. Any= change to any of those factors may cause the results to vary. You shou= ld consult other information and performance tests to assist you in ful= ly evaluating your contemplated purchases, including the performance of= that product when combined with other products.
>>>
>>> Thanks,
>>> Yingqi
>>>
>>>
>>>
>>> --
>>> Born in Roswell... married an alien...
>>> http://emptyhammock.com/
>>> http://edjective.org/
= >>>
>>>
>>>
>>>
>>> --
>>> Born in Roswell... married an alien...
>>> http://emptyhammock.com/
>>> http://edjective.org/
= >>>
>>> <httpd_trunk_so_reuseport.patch><httpd_trunk_buck= et.patch>
>>
>
> <httpd_trunk_so_reuseport.patch><httpd_trunk_bucket.patch= >


= --1__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938-- --0__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938 Content-type: image/gif; name="graycol.gif" Content-Disposition: inline; filename="graycol.gif" Content-ID: <1__=0ABBF67BDFDE29388f9e8a93df938@ca.ibm.com> Content-transfer-encoding: base64 R0lGODlhEAAQAKECAMzMzAAAAP///wAAACH5BAEAAAIALAAAAAAQABAAAAIXlI+py+0PopwxUbpu ZRfKZ2zgSJbmSRYAIf4fT3B0aW1pemVkIGJ5IFVsZWFkIFNtYXJ0U2F2ZXIhAAA7 --0__=0ABBF67BDFDE29388f9e8a93df938690918c0ABBF67BDFDE2938--