Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (athena.apache.org: domain of yingqi.lu@intel.com
 designates 143.182.124.21 as permitted sender)
From: "Lu, Yingqi" <yingqi.lu@intel.com>
To: "dev@httpd.apache.org" <dev@httpd.apache.org>
Subject: RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT
 support
Thread-Topic: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT
 support
Thread-Index: AQHPfyq27qvWERbyoE+NvktU3/tNWZtf5V+AgAATSwD//4/6IIAB0QoA
Date: Wed, 4 Jun 2014 19:39:50 +0000
Message-ID: 
 <9ACD5B67AAC5594CB6268234CF29CF9AA376392C@ORSMSX113.amr.corp.intel.com>
References: 
 <9ACD5B67AAC5594CB6268234CF29CF9AA37532AE@ORSMSX113.amr.corp.intel.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3757792@ORSMSX113.amr.corp.intel.com>
 <E8907DF1-72CE-439E-B3AA-131E3AB6D40B@jaguNET.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3757D88@ORSMSX113.amr.corp.intel.com>
 <3AB38FA5-CA8D-4818-8DA7-8198AE01A8BB@jaguNET.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3758EA6@ORSMSX113.amr.corp.intel.com>
 <D145CD25-F385-495E-BBFC-92240D777854@jaguNET.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA37610E3@ORSMSX113.amr.corp.intel.com>
 <99688D42-DC68-4159-9ACB-D24C4A3ECD15@jaguNET.com>,<A6218B2F-3F78-43EC-9186-7ABFEA280E96@jaguNET.com>
 <46E15AF1-6E0F-43B5-83A8-B0BDD6436824@intel.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3761DF8@ORSMSX113.amr.corp.intel.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3762528@ORSMSX113.amr.corp.intel.com>
 <68D58159-9AA8-4597-937E-D6B6B23C1531@jaguNET.com>
 <D8167D13-996A-40FA-8EF7-281E71507118@jaguNET.com>
 <A0B51374-6B09-48D8-BAA2-14E1FFC6E8CC@jaguNET.com>
 <9ACD5B67AAC5594CB6268234CF29CF9AA3762FF9@ORSMSX113.amr.corp.intel.com>
In-Reply-To: 
 <9ACD5B67AAC5594CB6268234CF29CF9AA3762FF9@ORSMSX113.amr.corp.intel.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/mixed;
	boundary="_002_9ACD5B67AAC5594CB6268234CF29CF9AA376392CORSMSX113amrcor_"
MIME-Version: 1.0

--_002_9ACD5B67AAC5594CB6268234CF29CF9AA376392CORSMSX113amrcor_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Jim,

I just found that prefork and worker has issue with restart. Event mpm code=
 is good.=20

I created this small patch to fix the issue for both prefork and worker. Th=
e patch is based on rev #1600451.

Can you please help add the changes in the trunk?

Thanks,
Yingqi

-----Original Message-----
From: Lu, Yingqi [mailto:yingqi.lu@intel.com]=20
Sent: Tuesday, June 03, 2014 8:50 AM
To: dev@httpd.apache.org
Subject: RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT=
 support

Thank you very much for your help!

Thanks,
Yingqi

-----Original Message-----
From: Jim Jagielski [mailto:jim@jaguNET.com]
Sent: Tuesday, June 03, 2014 8:31 AM
To: dev@httpd.apache.org
Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT=
 support

Next on the agenda is to push into eventopt

On Jun 3, 2014, at 10:21 AM, Jim Jagielski <jim@jaguNET.com> wrote:

> FTR: I saw no reason to try to handle both patches... I used the=20
> so_reuseport patch as the primary patch to focus on.
>=20
> I have some minor changes coming up to follow-up post the initial=20
> commit
>=20
> On Jun 3, 2014, at 8:51 AM, Jim Jagielski <jim@jagunet.com> wrote:
>=20
>> I have folded this into trunk and am currently fixing some compile=20
>> warnings and errors...
>>=20
>> On Jun 2, 2014, at 4:22 AM, Lu, Yingqi <yingqi.lu@intel.com> wrote:
>>=20
>>> Hi Jim,
>>>=20
>>> Personally, I think the second approach is better, it keeps ap_mpm_pod_=
signal () and ap_mpm_pod_killpg () exactly as the original ones, only modif=
ies dummy_connection (). Please let me know if you have different opinions.
>>>=20
>>> Attached is the latest version of the two patches. They were both gener=
ated against trunk rev. 1598561. Please review them and let me know if ther=
e is anything missing.
>>>=20
>>> I already updated the Bugzilla database for the item 55897 and item 562=
79.
>>>=20
>>> Thanks,
>>> Yingqi
>>>=20
>>> -----Original Message-----
>>> From: Lu, Yingqi [mailto:yingqi.lu@intel.com]
>>> Sent: Saturday, May 31, 2014 11:48 PM
>>> To: dev@httpd.apache.org
>>> Subject: RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
>>> SO_REUSEPORT support
>>>=20
>>> Hi Jim,
>>>=20
>>> Regarding to your comment #2, yes, you are right, it should be ap_mpm_p=
od_killpg(pod, retained->max_daemons_limit, i). Thanks very much for catchi=
ng this.
>>>=20
>>> Regarding to your comment #1, the patch modifies the dummy_connection(a=
p_pod_t *pod) to be dummy_connection(ap_pod_t *pod, int child_bucket). Insi=
de the function, the reference listen statement is mpm_listen[child_bucket]=
. And, ap_mpm_pod_signal() calls dummy_connection().=20
>>>=20
>>> Can we just modify the return of ap_mpm_pod_signal() from dummy_connect=
ion(pod) to dummy_connection(pod, 0) and add ap_mpm_pod_signal_ex()?=20
>>>=20
>>> Or, if we need to keep ap_mpm_pod_signal() exactly as the original, I c=
an modify dummy_connection() to send dummy data via all the duplicated list=
en statements. Then, we do not need child_bucket as the input parameter for=
 dummy_connection(). In this case, we do not need adding ap_mpm_pod_signal_=
ex() too.
>>>=20
>>> I already tested the code for the above approaches and they both work.=
=20
>>>=20
>>> Please let me know which way you think is better. I can quickly send yo=
u an update for review.
>>>=20
>>> Thanks,
>>> Yingqi
>>>=20
>>> -----Original Message-----
>>> From: Lu, Yingqi
>>> Sent: Saturday, May 31, 2014 3:28 PM
>>> To: <dev@httpd.apache.org>
>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
>>> SO_REUSEPORT support
>>>=20
>>> Hi Jim,
>>>=20
>>> Thanks very much for your email! I will look into both of them and send=
 an update tonight!
>>>=20
>>> Thanks,
>>> Yingqi
>>>=20
>>>> On May 31, 2014, at 9:43 AM, "Jim Jagielski" <jim@jaguNET.com> wrote:
>>>>=20
>>>> I also see:
>>>>=20
>>>>      /* kill off the idle ones */
>>>> -        ap_mpm_pod_killpg(pod, retained->max_daemons_limit);
>>>> +        for (i =3D 0; i < num_buckets; i++) {
>>>> +            ap_mpm_pod_killpg(pod[i], i, retained->max_daemons_limit)=
;
>>>> +        }
>>>>=20
>>>>=20
>>>> Is that right? Why isn't it: ap_mpm_pod_killpg(pod, retained->max_daem=
ons_limit, i); ??
>>>>=20
>>>> /**
>>>> * Write data to the pipe-of-death, signalling that all child=20
>>>> process
>>>> * should die.
>>>> * @param pod The pipe-of-death to write to.
>>>> * @param num The number of child processes to kill
>>>> + * @param my_bucket the bucket that holds the dying child process.
>>>> */
>>>> -AP_DECLARE(void) ap_mpm_pod_killpg(ap_pod_t *pod, int num);
>>>> +AP_DECLARE(void) ap_mpm_pod_killpg(ap_pod_t *pod, int num, int=20
>>>> +child_bucket);
>>>>=20
>>>> Isn't 'num' the same in both implementation??
>>>>=20
>>>>> On May 31, 2014, at 12:03 PM, Jim Jagielski <jim@jagunet.com> wrote:
>>>>>=20
>>>>> Sorry I didn't catch this earlier:
>>>>>=20
>>>>> I see
>>>>>=20
>>>>> +++ httpd-trunk.new/include/mpm_common.h    2014-05-16 13:07:03.89298=
7491 -0400
>>>>> @@ -267,16 +267,18 @@
>>>>> * Write data to the pipe-of-death, signalling that one child=20
>>>>> process
>>>>> * should die.
>>>>> * @param pod the pipe-of-death to write to.
>>>>> + * @param my_bucket the bucket that holds the dying child process.
>>>>> */
>>>>> -AP_DECLARE(apr_status_t) ap_mpm_pod_signal(ap_pod_t *pod);
>>>>> +AP_DECLARE(apr_status_t) ap_mpm_pod_signal(ap_pod_t *pod, int=20
>>>>> +child_bucket);
>>>>>=20
>>>>> We can change the API at this point. We could add another=20
>>>>> function, eg ap_mpm_pod_signal_ex() which takes the int param, but=20
>>>>> we can't modify ap_mpm_pod_signal() itself.
>>>>>=20
>>>>>> On May 30, 2014, at 11:15 AM, Lu, Yingqi <yingqi.lu@intel.com> wrote=
:
>>>>>>=20
>>>>>> Thank you very much!
>>>>>>=20
>>>>>> Thanks,
>>>>>> Yingqi
>>>>>>=20
>>>>>> -----Original Message-----
>>>>>> From: Jim Jagielski [mailto:jim@jaguNET.com]
>>>>>> Sent: Friday, May 30, 2014 7:07 AM
>>>>>> To: dev@httpd.apache.org
>>>>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
>>>>>> SO_REUSEPORT support
>>>>>>=20
>>>>>> Thx! Let me review. My plan is to fold into trunk this weekend.
>>>>>>=20
>>>>>>> On May 16, 2014, at 2:53 PM, Lu, Yingqi <yingqi.lu@intel.com> wrote=
:
>>>>>>>=20
>>>>>>> Hi Jim,
>>>>>>>=20
>>>>>>> Thanks very much for clarifying this with me. I added #ifdef in the=
 code to check _SC_NPROCESSORS_ONLN in the so_reuseport patch. Bucket patch=
 does not use this parameter so that it remains the same.
>>>>>>>=20
>>>>>>> Attached are the two most recent patches. I already updated the bug=
zilla #55897 as well.
>>>>>>>=20
>>>>>>> Thanks,
>>>>>>> Yingqi
>>>>>>>=20
>>>>>>> -----Original Message-----
>>>>>>> From: Jim Jagielski [mailto:jim@jaguNET.com]
>>>>>>> Sent: Thursday, May 15, 2014 7:53 AM
>>>>>>> To: dev@httpd.apache.org
>>>>>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
>>>>>>> SO_REUSEPORT support
>>>>>>>=20
>>>>>>> I was thinking more about the sysconf(_SC_NPROCESSORS_ONLN) stuff..=
.
>>>>>>> We could either check for that during config/build or protect it wi=
th a #ifdef in the code (and create some logging so the admin nows if it wa=
s found or not).
>>>>>>>=20
>>>>>>>> On May 14, 2014, at 11:59 AM, Lu, Yingqi <yingqi.lu@intel.com> wro=
te:
>>>>>>>>=20
>>>>>>>> Hi Jim,
>>>>>>>>=20
>>>>>>>> Thanks very much for your email.
>>>>>>>>=20
>>>>>>>> In the SO_REUSEPORT patch, SO_REUSEPORT support is checked inside =
listen.c file. If the feature is not supported on the OS (for example, Linu=
x kernel < 3.9), it will fall back to the original behavior.=20
>>>>>>>>=20
>>>>>>>> In the bucket patch, there is no need to check the params. With si=
ngle listen statement, it is just the default behavior.=20
>>>>>>>>=20
>>>>>>>> Please let me know if this answers your question.
>>>>>>>>=20
>>>>>>>> Thanks,
>>>>>>>> Yingqi
>>>>>>>>=20
>>>>>>>> -----Original Message-----
>>>>>>>> From: Jim Jagielski [mailto:jim@jaguNET.com]
>>>>>>>> Sent: Wednesday, May 14, 2014 6:57 AM
>>>>>>>> To: dev@httpd.apache.org
>>>>>>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
>>>>>>>> SO_REUSEPORT support
>>>>>>>>=20
>>>>>>>> This is very cool!
>>>>>>>>=20
>>>>>>>> mod_status assumes that sysconf() exists, but do we need to do a c=
onfig check on the params we use in these patches?
>>>>>>>> We look OK on Linux, FreeBSD and OSX...
>>>>>>>>=20
>>>>>>>> I'm +1 on folding into trunk.
>>>>>>>>=20
>>>>>>>>> On May 13, 2014, at 7:55 PM, Lu, Yingqi <yingqi.lu@intel.com> wro=
te:
>>>>>>>>>=20
>>>>>>>>> Dear All,
>>>>>>>>>=20
>>>>>>>>> During the last couple weeks, I spent some time extending the ori=
ginal two patches from prefork MPM only to all three Linux MPMs (prefork, w=
orker and event). Attached is the latest version of the two patches. Bugzil=
la database has also been updated already. The ID for the two patches are #=
55897 and #56279. Please refer to messages below for details on both of the=
 patches.
>>>>>>>>>=20
>>>>>>>>> Quick test result on modern dual socket Intel platform (Linux=20
>>>>>>>>> Kernel
>>>>>>>>> 3.13.9) SO_REUSEPORT patch (bugzilla #55897)
>>>>>>>>> 1.       Prefork MPM: 1 listen statement: 2.16X throughput improv=
ement; 2 listen statements: 2.33X throughput improvement
>>>>>>>>> 2.       Worker MPM: 1 listen statement: 10% throughput improveme=
nt; 2 listen statements: 35% throughput improvement
>>>>>>>>> 3.       Event MPM: 1 listen statement: 13% throughput improvemen=
t; 2 listen statements: throughput parity, but 62% response time reduction =
(with patch, 38% response time as original SW)
>>>>>>>>>=20
>>>>>>>>> Bucket patch (bugzilla #56279, only impact multiple listen statem=
ent case)
>>>>>>>>> 1.       Prefork MPM: 2 listen statements: 42% throughput improve=
ment
>>>>>>>>> 2.       Worker MPM: 2 listen statements: 7% throughput improveme=
nt
>>>>>>>>>=20
>>>>>>>>> In all the above testing cases, significant response time reducti=
ons are observed, even with throughput improvements.
>>>>>>>>>=20
>>>>>>>>> Please let me know your feedback and comments.
>>>>>>>>>=20
>>>>>>>>> Thanks,
>>>>>>>>> Yingqi
>>>>>>>>> Software and workloads used in performance tests may have been op=
timized for performance only on Intel microprocessors. Performance tests, s=
uch as SYSmark and MobileMark, are measured using specific computer systems=
, components, software, operations and functions. Any change to any of thos=
e factors may cause the results to vary. You should consult other informati=
on and performance tests to assist you in fully evaluating your contemplate=
d purchases, including the performance of that product when combined with o=
ther products.
>>>>>>>>>=20
>>>>>>>>> From: Lu, Yingqi [mailto:yingqi.lu@intel.com]
>>>>>>>>> Sent: Monday, March 17, 2014 1:41 PM
>>>>>>>>> To: dev@httpd.apache.org
>>>>>>>>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch=20
>>>>>>>>> with SO_REUSEPORT support
>>>>>>>>>=20
>>>>>>>>> Dear all,
>>>>>>>>>=20
>>>>>>>>> Based on the feedback we received, we modified this patch. Here i=
s the most recent version. We also modified the Bugzilla database(Bugzilla#=
 55897 for SO_REUSEPORT patch; Bugzilla# 56279 for bucket patch).
>>>>>>>>>=20
>>>>>>>>> Below are the changes we made into this new version:
>>>>>>>>>=20
>>>>>>>>> According to Yann Ylavic and other people's comments, we separate=
 the original patch between with and without SO_REUSEPORT into two separate=
d patches. The SO_REUSEPORT patch does not change the original listen socke=
ts, it just duplicate the original one into multiple ones. Since the listen=
 sockets are identical, there is no need to change the idle_server_maintena=
nce function. The bucket patch (without SO_REUSEPORT), on the other hand, i=
t breaks down the original listen record (if there are multiple listen sock=
s) to multiple listen record linked lists. In this case, idle_server_mainte=
nance is implemented at bucket level to address the situation that imbalanc=
ed traffic occurs among different listen sockets/children buckets. In the b=
ucket patch, the polling in the child process is removed since each child o=
nly listens to 1 sock.
>>>>>>>>>=20
>>>>>>>>> According to Arkadiusz Miskiewicz's comment, we make the "detecti=
on of SO_REUSEPORT" at run time.
>>>>>>>>>=20
>>>>>>>>> According to Jeff Trawick's comments, 1. We generate the=20
>>>>>>>>> patches against the httpd trunk.
>>>>>>>>> 2. We tested the current patches and they do not impact event and=
 worker mpms. If current patches can be accepted, we would be happy to exte=
nd them to other Linux based mpms. There are not much code changes, but req=
uire some time to setup the workload to test.
>>>>>>>>> 3. We removed unnecessary comments and changed APLOGNO(). We also=
 changed some of the parameter/variable/function names to better represent =
their meanings.
>>>>>>>>> 4. There should be no build-in limitations for SO_REUSEPORT patch=
. For bucket patch, the only thing is the number of children bucket only sc=
ales to MAX_SPAWN_RATE. If there are more than 32 (current default MAX_SPQN=
_RATE) listen statements specified in the httpd.conf, the number of buckets=
 will be fixed to 32. The reason for this is because that we implement the =
idle_server_maintenance at bucket level, each bucket's own max_spawn_rate i=
s set to MAX_SPAWN_RATE/num_buckets.
>>>>>>>>>=20
>>>>>>>>> Again, thanks very much for all the comments and feedback. Please=
 let us know if there are more changes we need to complete to make them acc=
epted.
>>>>>>>>>=20
>>>>>>>>> Thanks,
>>>>>>>>> Yingqi Lu
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> From: Lu, Yingqi
>>>>>>>>> Sent: Tuesday, March 04, 2014 10:43 AM
>>>>>>>>> To: dev@httpd.apache.org
>>>>>>>>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch=20
>>>>>>>>> with SO_REUSEPORT support
>>>>>>>>>=20
>>>>>>>>> Hi Jeff,
>>>>>>>>>=20
>>>>>>>>> Thanks very much for your time reviewing the patch! We will modif=
y the patch according to your comments and repost it here.
>>>>>>>>>=20
>>>>>>>>> Thanks,
>>>>>>>>> Yingqi
>>>>>>>>>=20
>>>>>>>>> From: Jeff Trawick [mailto:trawick@gmail.com]
>>>>>>>>> Sent: Tuesday, March 04, 2014 10:08 AM
>>>>>>>>> To: Apache HTTP Server Development List
>>>>>>>>> Subject: Re: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch=20
>>>>>>>>> with SO_REUSEPORT support
>>>>>>>>>=20
>>>>>>>>> On Tue, Mar 4, 2014 at 10:35 AM, Lu, Yingqi <yingqi.lu@intel.com>=
 wrote:
>>>>>>>>> Hi All,
>>>>>>>>>=20
>>>>>>>>> I just want to ping again on this patch to gather your feedback a=
nd comments. Please refer to the messages below for patch details.
>>>>>>>>>=20
>>>>>>>>> If you need any additional information/supporting data, please le=
t us know as well.
>>>>>>>>>=20
>>>>>>>>> Yeah, it has been on my todo list, but I don't have time to=20
>>>>>>>>> give an in depth review at the moment.  Here are a few questions/=
comments.
>>>>>>>>> (And you'll have to deal with the fact that it is=20
>>>>>>>>> unnecessarily tedious for me to evaluate higher-level=20
>>>>>>>>> considerations if there are a lot of distractions, such as the=20
>>>>>>>>> code comments below ;) But others are of course free to chime
>>>>>>>>> in.)
>>>>>>>>>=20
>>>>>>>>> The patch should be against httpd trunk.  It probably won't take =
much time for you to create that patch and confirm basic operation.
>>>>>>>>>=20
>>>>>>>>> What is the impact to other MPMs, even if they shouldn't use or d=
on't have the necessary code to use SO_REUSEPORT at this time?
>>>>>>>>>=20
>>>>>>>>> Have you tried the event MPM?
>>>>>>>>>=20
>>>>>>>>> Is there a way for the admin to choose this behavior?  Most won't=
 care, but everyone's behavior is changed AFAICT.
>>>>>>>>>=20
>>>>>>>>> Are there built-in limitations in this patch that we should be aw=
are of?  E.g., the free slot/spawn rate changes suggest to me that there ca=
n't be more than 1025 children???
>>>>>>>>>=20
>>>>>>>>> We should assume for now that there's no reason this couldn't be =
committed to trunk after review/rework, so make sure it is as close as you =
can get it to what you think is the final form.
>>>>>>>>>=20
>>>>>>>>> For the configure-time check for 3.9 kernel: I think we'd also=20
>>>>>>>>> use AC_TRY_COMPILE at configure time to confirm that the=20
>>>>>>>>> SO_REUSEPORT definition is available, and not enable it if the=20
>>>>>>>>> system includes doesn't define it.  (Does that cause a problem=20
>>>>>>>>> for any significant number of people?)
>>>>>>>>>=20
>>>>>>>>> Don't mention the patch in the patch ;) (e.g., "This function=20
>>>>>>>>> is added for the patch.")
>>>>>>>>>=20
>>>>>>>>> Incomplete comments on style/syntax issues:
>>>>>>>>>=20
>>>>>>>>> * mixing declarations and statements (e.g., "duplr->next =3D 0;=20
>>>>>>>>> apr_socket_t *temps;") isn't supported by all compilers and is=20
>>>>>>>>> distracting when reviewing
>>>>>>>>> * suitable identifier names (e.g., fix global variable "flag"=20
>>>>>>>>> and whatever else isn't appropriate; "ap_post_config_listeners"
>>>>>>>>> should be renamed to indicate what it does
>>>>>>>>> * APLOGNO(99999) and comments about fixing it: Instead put "APLOG=
NO()"=20
>>>>>>>>> and don't add reminders in comments
>>>>>>>>> * this doesn't seem portable: "int free_slots[MAX_SPAWN_RATE/num_=
buckets];"
>>>>>>>>> and so on
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> Thanks,
>>>>>>>>> Yingqi
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> From: Lu, Yingqi
>>>>>>>>> Sent: Friday, January 24, 2014 3:26 PM
>>>>>>>>> To: dev@httpd.apache.org
>>>>>>>>> Subject: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20
>>>>>>>>> SO_REUSEPORT support
>>>>>>>>>=20
>>>>>>>>> Dear All,
>>>>>>>>>=20
>>>>>>>>> Our analysis of Apache httpd 2.4.7 prefork mpm, on 32 and 64 thre=
ad Intel Xeon 2600 series systems, using an open source three tier social n=
etworking web server workload, revealed performance scaling issues.  In cur=
rent software single listen statement (listen 80) provides better scalabili=
ty due to un-serialized accept. However, when system is under very high loa=
d, this can lead to big number of child processes stuck in D state. On the =
other hand, the serialized accept approach cannot scale with the high load =
either.  In our analysis, a 32-thread system, with 2 listen statements spec=
ified, could scale to just 70% utilization, and a 64-thread system, with si=
gnal listen statement specified (listen 80, 4 network interfaces), could sc=
ale to only 60% utilization.
>>>>>>>>>=20
>>>>>>>>> Based on those findings, we created a prototype patch for prefork=
 mpm which extends performance and thread utilization. In Linux kernel newe=
r than 3.9, SO_REUSEPORT is enabled. This feature allows multiple sockets l=
isten to the same IP:port and automatically round robins connections. We us=
e this feature to create multiple duplicated listener records of the origin=
al one and partition the child processes into buckets. Each bucket listens =
to 1 IP:port. In case of old kernel which does not have the SO_REUSEPORT en=
abled, we modified the "multiple listen statement case" by creating 1 liste=
n record for each listen statement and partitioning the child processes int=
o different buckets. Each bucket listens to 1 IP:port.
>>>>>>>>>=20
>>>>>>>>> Quick tests of the patch, running the same workload, demonstrated=
 a 22% throughput increase with 32-threads system and 2 listen statements (=
Linux kernel 3.10.4). With the older kernel (Linux Kernel 3.8.8, without SO=
_REUSEPORT), 10% performance gain was measured. With single listen statemen=
t (listen 80) configuration, we observed over 2X performance improvements o=
n modern dual socket Intel platforms (Linux Kernel 3.10.4). We also observe=
d big reduction in response time, in addition to the throughput improvement=
 gained in our tests 1.
>>>>>>>>>=20
>>>>>>>>> Following the feedback from the bugzilla website where we origina=
lly submitted the patch, we removed the dependency of APR change to simplif=
y the patch testing process. Thanks Jeff Trawick for his good suggestion! W=
e are also actively working on extending the patch to worker and event MPMs=
, as a next step. Meanwhile, we would like to gather comments from all of y=
ou on the current prefork patch. Please take some time test it and let us k=
now how it works in your environment.
>>>>>>>>>=20
>>>>>>>>> This is our first patch to the Apache community. Please help us r=
eview it and let us know if there is anything we might revise to improve it=
. Your feedback is very much appreciated.
>>>>>>>>>=20
>>>>>>>>> Configuration:
>>>>>>>>> <IfModule prefork.c>
>>>>>>>>> ListenBacklog 105384
>>>>>>>>> ServerLimit 105000
>>>>>>>>> MaxClients 1024
>>>>>>>>> MaxRequestsPerChild 0
>>>>>>>>> StartServers 64
>>>>>>>>> MinSpareServers 8
>>>>>>>>> MaxSpareServers 16
>>>>>>>>> </IfModule>
>>>>>>>>>=20
>>>>>>>>> 1. Software and workloads used in performance tests may have been=
 optimized for performance only on Intel microprocessors. Performance tests=
, such as SYSmark and MobileMark, are measured using specific computer syst=
ems, components, software, operations and functions. Any change to any of t=
hose factors may cause the results to vary. You should consult other inform=
ation and performance tests to assist you in fully evaluating your contempl=
ated purchases, including the performance of that product when combined wit=
h other products.
>>>>>>>>>=20
>>>>>>>>> Thanks,
>>>>>>>>> Yingqi
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> --
>>>>>>>>> Born in Roswell... married an alien...
>>>>>>>>> http://emptyhammock.com/
>>>>>>>>> http://edjective.org/
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> --
>>>>>>>>> Born in Roswell... married an alien...
>>>>>>>>> http://emptyhammock.com/
>>>>>>>>> http://edjective.org/
>>>>>>>>>=20
>>>>>>>>> <httpd_trunk_so_reuseport.patch><httpd_trunk_bucket.patch>
>>>>>>>=20
>>>>>>> <httpd_trunk_so_reuseport.patch><httpd_trunk_bucket.patch>
>>>>=20
>>> <httpd_trunk_so_reuseport.patch><httpd_trunk_bucket.patch>
>>=20
>=20


--_002_9ACD5B67AAC5594CB6268234CF29CF9AA376392CORSMSX113amrcor_
Content-Type: application/octet-stream; name="restart_fix.patch"
Content-Description: restart_fix.patch
Content-Disposition: attachment; filename="restart_fix.patch"; size=1036;
	creation-date="Wed, 04 Jun 2014 19:29:39 GMT";
	modification-date="Wed, 04 Jun 2014 19:30:46 GMT"
Content-Transfer-Encoding: base64

ZGlmZiAtcnUgaHR0cGQtdHJ1bmsvc2VydmVyL21wbS9wcmVmb3JrL3ByZWZvcmsuYyBodHRwZC10
cnVuay5uZXcvc2VydmVyL21wbS9wcmVmb3JrL3ByZWZvcmsuYwotLS0gaHR0cGQtdHJ1bmsvc2Vy
dmVyL21wbS9wcmVmb3JrL3ByZWZvcmsuYwkyMDE0LTA2LTAzIDE0OjI2OjQwLjE5MDk3NjU2NSAt
MDQwMAorKysgaHR0cGQtdHJ1bmsubmV3L3NlcnZlci9tcG0vcHJlZm9yay9wcmVmb3JrLmMJMjAx
NC0wNi0wNCAxNDoxODoyMy4yNjc1NTI4MjUgLTA0MDAKQEAgLTk3MSw5ICs5NzEsNiBAQAogICAg
ICAgICAgICAgcmV0dXJuIERPTkU7CiAgICAgICAgIH0KICAgICAgfQotICAgIGZvciAobHIgPSBh
cF9saXN0ZW5lcnM7IGxyOyBsciA9IGxyLT5uZXh0KSB7Ci0gICAgICAgIGFwcl9zb2NrZXRfY2xv
c2UobHItPnNkKTsKLSAgICB9CiAKICAgICBpZiAoIXJldGFpbmVkLT5pc19ncmFjZWZ1bCkgewog
ICAgICAgICBpZiAoYXBfcnVuX3ByZV9tcG0ocy0+cHJvY2Vzcy0+cG9vbCwgU0JfU0hBUkVEKSAh
PSBPSykgewpkaWZmIC1ydSBodHRwZC10cnVuay9zZXJ2ZXIvbXBtL3dvcmtlci93b3JrZXIuYyBo
dHRwZC10cnVuay5uZXcvc2VydmVyL21wbS93b3JrZXIvd29ya2VyLmMKLS0tIGh0dHBkLXRydW5r
L3NlcnZlci9tcG0vd29ya2VyL3dvcmtlci5jCTIwMTQtMDYtMDMgMTQ6MjY6NDAuMzA1OTc2NTYz
IC0wNDAwCisrKyBodHRwZC10cnVuay5uZXcvc2VydmVyL21wbS93b3JrZXIvd29ya2VyLmMJMjAx
NC0wNi0wNCAxNDoxODozOS4yMTk1NTI1NzAgLTA0MDAKQEAgLTE4MTYsOSArMTgxNiw2IEBACiAg
ICAgICAgICAgICByZXR1cm4gRE9ORTsKICAgICAgICAgfQogICAgIH0KLSAgICBmb3IgKGxyID0g
YXBfbGlzdGVuZXJzOyBscjsgbHIgPSBsci0+bmV4dCkgewotICAgICAgICBhcHJfc29ja2V0X2Ns
b3NlKGxyLT5zZCk7Ci0gICAgIH0KIAogICAgIGlmICghcmV0YWluZWQtPmlzX2dyYWNlZnVsKSB7
CiAgICAgICAgIGlmIChhcF9ydW5fcHJlX21wbShzLT5wcm9jZXNzLT5wb29sLCBTQl9TSEFSRUQp
ICE9IE9LKSB7Cg==

--_002_9ACD5B67AAC5594CB6268234CF29CF9AA376392CORSMSX113amrcor_--