Return-Path: X-Original-To: apmail-httpd-dev-archive@www.apache.org Delivered-To: apmail-httpd-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0F678C52D for ; Thu, 5 Jun 2014 13:38:58 +0000 (UTC) Received: (qmail 63283 invoked by uid 500); 5 Jun 2014 13:38:57 -0000 Delivered-To: apmail-httpd-dev-archive@httpd.apache.org Received: (qmail 63210 invoked by uid 500); 5 Jun 2014 13:38:57 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 63202 invoked by uid 99); 5 Jun 2014 13:38:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2014 13:38:57 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [76.96.30.80] (HELO qmta08.emeryville.ca.mail.comcast.net) (76.96.30.80) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2014 13:38:52 +0000 Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43]) by qmta08.emeryville.ca.mail.comcast.net with comcast id Ad9D1o0020vp7WLA8deYc4; Thu, 05 Jun 2014 13:38:32 +0000 Received: from [192.168.199.10] ([69.251.80.74]) by omta05.emeryville.ca.mail.comcast.net with comcast id AdeV1o00A1cCKD98RdeWy2; Thu, 05 Jun 2014 13:38:31 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.2\)) Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support From: Jim Jagielski In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA376392C@ORSMSX113.amr.corp.intel.com> Date: Thu, 5 Jun 2014 09:38:28 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <9ACD5B67AAC5594CB6268234CF29CF9AA37532AE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3757792@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3757D88@ORSMSX113.amr.corp.intel.com> <3AB38FA5-CA8D-4818-8DA7-8198AE01A8BB@jaguNET.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3758EA6@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA37610E3@ORSMSX113.amr.corp.intel.com> <99688D42-DC68-4159-9ACB-D24C4A3ECD15@jaguNET.com>, <46E15AF1-6E0F-43B5-83A8-B0BDD6436824@intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3761DF8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA3762528@ORSMSX113.amr.corp.intel.com> <68D58159-9AA8-4597-937E-D6B6B23C1531@jaguNET.com> <9ACD5B67AAC5594CB6268234 CF29CF9AA3762FF9@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA376392C@ORSMSX113.amr.corp.intel.com> To: dev@httpd.apache.org X-Mailer: Apple Mail (2.1878.2) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20140121; t=1401975512; bh=WKagdO6XR1LFbpX8FEPQY0jJzg+Z8xZEpShkDjyQiL8=; h=Received:Received:Content-Type:Mime-Version:Subject:From:Date: Message-Id:To; b=NyzEwtyV8D/6VOA2kZcpRwauZuyiq9q6D1l6Tc2Wvc/Yk38DZ18FZn2VOce9xRvp0 bgQHYT/NThlO1KjD8Cs1eYFqSM3iiQRaIlv7JQ+B0/KXQVXym7LskML+S17XFHUVrz MGOIQtRwgcmOLojUUg4eW2+4JSeCDnY1COdz1JOHzPSYKsz6KZDCk39T24mWTpkgzu xGTKqpFVrkKMX8qwfSxLP7PX1skYkTO+yoAz3JY3hIjurTIJ4szIrb+NajBbWvVBjE iytBZB9C+ODzGa9Y7W6TJbp3ReqERuC88u+YfHlv/93BZrdpoI+AtL7sh29vhWWFGQ 2qaflsbBaKY5Q== X-Virus-Checked: Checked by ClamAV on apache.org Committed r1600656 Thx On Jun 4, 2014, at 3:39 PM, Lu, Yingqi wrote: > Hi Jim, >=20 > I just found that prefork and worker has issue with restart. Event mpm = code is good.=20 >=20 > I created this small patch to fix the issue for both prefork and = worker. The patch is based on rev #1600451. >=20 > Can you please help add the changes in the trunk? >=20 > Thanks, > Yingqi >=20 > -----Original Message----- > From: Lu, Yingqi [mailto:yingqi.lu@intel.com]=20 > Sent: Tuesday, June 03, 2014 8:50 AM > To: dev@httpd.apache.org > Subject: RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with = SO_REUSEPORT support >=20 > Thank you very much for your help! >=20 > Thanks, > Yingqi >=20 > -----Original Message----- > From: Jim Jagielski [mailto:jim@jaguNET.com] > Sent: Tuesday, June 03, 2014 8:31 AM > To: dev@httpd.apache.org > Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with = SO_REUSEPORT support >=20 > Next on the agenda is to push into eventopt >=20 > On Jun 3, 2014, at 10:21 AM, Jim Jagielski wrote: >=20 >> FTR: I saw no reason to try to handle both patches... I used the=20 >> so_reuseport patch as the primary patch to focus on. >>=20 >> I have some minor changes coming up to follow-up post the initial=20 >> commit >>=20 >> On Jun 3, 2014, at 8:51 AM, Jim Jagielski wrote: >>=20 >>> I have folded this into trunk and am currently fixing some compile=20= >>> warnings and errors... >>>=20 >>> On Jun 2, 2014, at 4:22 AM, Lu, Yingqi wrote: >>>=20 >>>> Hi Jim, >>>>=20 >>>> Personally, I think the second approach is better, it keeps = ap_mpm_pod_signal () and ap_mpm_pod_killpg () exactly as the original = ones, only modifies dummy_connection (). Please let me know if you have = different opinions. >>>>=20 >>>> Attached is the latest version of the two patches. They were both = generated against trunk rev. 1598561. Please review them and let me know = if there is anything missing. >>>>=20 >>>> I already updated the Bugzilla database for the item 55897 and item = 56279. >>>>=20 >>>> Thanks, >>>> Yingqi >>>>=20 >>>> -----Original Message----- >>>> From: Lu, Yingqi [mailto:yingqi.lu@intel.com] >>>> Sent: Saturday, May 31, 2014 11:48 PM >>>> To: dev@httpd.apache.org >>>> Subject: RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20 >>>> SO_REUSEPORT support >>>>=20 >>>> Hi Jim, >>>>=20 >>>> Regarding to your comment #2, yes, you are right, it should be = ap_mpm_pod_killpg(pod, retained->max_daemons_limit, i). Thanks very much = for catching this. >>>>=20 >>>> Regarding to your comment #1, the patch modifies the = dummy_connection(ap_pod_t *pod) to be dummy_connection(ap_pod_t *pod, = int child_bucket). Inside the function, the reference listen statement = is mpm_listen[child_bucket]. And, ap_mpm_pod_signal() calls = dummy_connection().=20 >>>>=20 >>>> Can we just modify the return of ap_mpm_pod_signal() from = dummy_connection(pod) to dummy_connection(pod, 0) and add = ap_mpm_pod_signal_ex()?=20 >>>>=20 >>>> Or, if we need to keep ap_mpm_pod_signal() exactly as the original, = I can modify dummy_connection() to send dummy data via all the = duplicated listen statements. Then, we do not need child_bucket as the = input parameter for dummy_connection(). In this case, we do not need = adding ap_mpm_pod_signal_ex() too. >>>>=20 >>>> I already tested the code for the above approaches and they both = work.=20 >>>>=20 >>>> Please let me know which way you think is better. I can quickly = send you an update for review. >>>>=20 >>>> Thanks, >>>> Yingqi >>>>=20 >>>> -----Original Message----- >>>> From: Lu, Yingqi >>>> Sent: Saturday, May 31, 2014 3:28 PM >>>> To: >>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20 >>>> SO_REUSEPORT support >>>>=20 >>>> Hi Jim, >>>>=20 >>>> Thanks very much for your email! I will look into both of them and = send an update tonight! >>>>=20 >>>> Thanks, >>>> Yingqi >>>>=20 >>>>> On May 31, 2014, at 9:43 AM, "Jim Jagielski" = wrote: >>>>>=20 >>>>> I also see: >>>>>=20 >>>>> /* kill off the idle ones */ >>>>> - ap_mpm_pod_killpg(pod, retained->max_daemons_limit); >>>>> + for (i =3D 0; i < num_buckets; i++) { >>>>> + ap_mpm_pod_killpg(pod[i], i, = retained->max_daemons_limit); >>>>> + } >>>>>=20 >>>>>=20 >>>>> Is that right? Why isn't it: ap_mpm_pod_killpg(pod, = retained->max_daemons_limit, i); ?? >>>>>=20 >>>>> /** >>>>> * Write data to the pipe-of-death, signalling that all child=20 >>>>> process >>>>> * should die. >>>>> * @param pod The pipe-of-death to write to. >>>>> * @param num The number of child processes to kill >>>>> + * @param my_bucket the bucket that holds the dying child = process. >>>>> */ >>>>> -AP_DECLARE(void) ap_mpm_pod_killpg(ap_pod_t *pod, int num); >>>>> +AP_DECLARE(void) ap_mpm_pod_killpg(ap_pod_t *pod, int num, int=20 >>>>> +child_bucket); >>>>>=20 >>>>> Isn't 'num' the same in both implementation?? >>>>>=20 >>>>>> On May 31, 2014, at 12:03 PM, Jim Jagielski = wrote: >>>>>>=20 >>>>>> Sorry I didn't catch this earlier: >>>>>>=20 >>>>>> I see >>>>>>=20 >>>>>> +++ httpd-trunk.new/include/mpm_common.h 2014-05-16 = 13:07:03.892987491 -0400 >>>>>> @@ -267,16 +267,18 @@ >>>>>> * Write data to the pipe-of-death, signalling that one child=20 >>>>>> process >>>>>> * should die. >>>>>> * @param pod the pipe-of-death to write to. >>>>>> + * @param my_bucket the bucket that holds the dying child = process. >>>>>> */ >>>>>> -AP_DECLARE(apr_status_t) ap_mpm_pod_signal(ap_pod_t *pod); >>>>>> +AP_DECLARE(apr_status_t) ap_mpm_pod_signal(ap_pod_t *pod, int=20 >>>>>> +child_bucket); >>>>>>=20 >>>>>> We can change the API at this point. We could add another=20 >>>>>> function, eg ap_mpm_pod_signal_ex() which takes the int param, = but=20 >>>>>> we can't modify ap_mpm_pod_signal() itself. >>>>>>=20 >>>>>>> On May 30, 2014, at 11:15 AM, Lu, Yingqi = wrote: >>>>>>>=20 >>>>>>> Thank you very much! >>>>>>>=20 >>>>>>> Thanks, >>>>>>> Yingqi >>>>>>>=20 >>>>>>> -----Original Message----- >>>>>>> From: Jim Jagielski [mailto:jim@jaguNET.com] >>>>>>> Sent: Friday, May 30, 2014 7:07 AM >>>>>>> To: dev@httpd.apache.org >>>>>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20= >>>>>>> SO_REUSEPORT support >>>>>>>=20 >>>>>>> Thx! Let me review. My plan is to fold into trunk this weekend. >>>>>>>=20 >>>>>>>> On May 16, 2014, at 2:53 PM, Lu, Yingqi = wrote: >>>>>>>>=20 >>>>>>>> Hi Jim, >>>>>>>>=20 >>>>>>>> Thanks very much for clarifying this with me. I added #ifdef in = the code to check _SC_NPROCESSORS_ONLN in the so_reuseport patch. Bucket = patch does not use this parameter so that it remains the same. >>>>>>>>=20 >>>>>>>> Attached are the two most recent patches. I already updated the = bugzilla #55897 as well. >>>>>>>>=20 >>>>>>>> Thanks, >>>>>>>> Yingqi >>>>>>>>=20 >>>>>>>> -----Original Message----- >>>>>>>> From: Jim Jagielski [mailto:jim@jaguNET.com] >>>>>>>> Sent: Thursday, May 15, 2014 7:53 AM >>>>>>>> To: dev@httpd.apache.org >>>>>>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20= >>>>>>>> SO_REUSEPORT support >>>>>>>>=20 >>>>>>>> I was thinking more about the sysconf(_SC_NPROCESSORS_ONLN) = stuff... >>>>>>>> We could either check for that during config/build or protect = it with a #ifdef in the code (and create some logging so the admin nows = if it was found or not). >>>>>>>>=20 >>>>>>>>> On May 14, 2014, at 11:59 AM, Lu, Yingqi = wrote: >>>>>>>>>=20 >>>>>>>>> Hi Jim, >>>>>>>>>=20 >>>>>>>>> Thanks very much for your email. >>>>>>>>>=20 >>>>>>>>> In the SO_REUSEPORT patch, SO_REUSEPORT support is checked = inside listen.c file. If the feature is not supported on the OS (for = example, Linux kernel < 3.9), it will fall back to the original = behavior.=20 >>>>>>>>>=20 >>>>>>>>> In the bucket patch, there is no need to check the params. = With single listen statement, it is just the default behavior.=20 >>>>>>>>>=20 >>>>>>>>> Please let me know if this answers your question. >>>>>>>>>=20 >>>>>>>>> Thanks, >>>>>>>>> Yingqi >>>>>>>>>=20 >>>>>>>>> -----Original Message----- >>>>>>>>> From: Jim Jagielski [mailto:jim@jaguNET.com] >>>>>>>>> Sent: Wednesday, May 14, 2014 6:57 AM >>>>>>>>> To: dev@httpd.apache.org >>>>>>>>> Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20= >>>>>>>>> SO_REUSEPORT support >>>>>>>>>=20 >>>>>>>>> This is very cool! >>>>>>>>>=20 >>>>>>>>> mod_status assumes that sysconf() exists, but do we need to do = a config check on the params we use in these patches? >>>>>>>>> We look OK on Linux, FreeBSD and OSX... >>>>>>>>>=20 >>>>>>>>> I'm +1 on folding into trunk. >>>>>>>>>=20 >>>>>>>>>> On May 13, 2014, at 7:55 PM, Lu, Yingqi = wrote: >>>>>>>>>>=20 >>>>>>>>>> Dear All, >>>>>>>>>>=20 >>>>>>>>>> During the last couple weeks, I spent some time extending the = original two patches from prefork MPM only to all three Linux MPMs = (prefork, worker and event). Attached is the latest version of the two = patches. Bugzilla database has also been updated already. The ID for the = two patches are #55897 and #56279. Please refer to messages below for = details on both of the patches. >>>>>>>>>>=20 >>>>>>>>>> Quick test result on modern dual socket Intel platform (Linux=20= >>>>>>>>>> Kernel >>>>>>>>>> 3.13.9) SO_REUSEPORT patch (bugzilla #55897) >>>>>>>>>> 1. Prefork MPM: 1 listen statement: 2.16X throughput = improvement; 2 listen statements: 2.33X throughput improvement >>>>>>>>>> 2. Worker MPM: 1 listen statement: 10% throughput = improvement; 2 listen statements: 35% throughput improvement >>>>>>>>>> 3. Event MPM: 1 listen statement: 13% throughput = improvement; 2 listen statements: throughput parity, but 62% response = time reduction (with patch, 38% response time as original SW) >>>>>>>>>>=20 >>>>>>>>>> Bucket patch (bugzilla #56279, only impact multiple listen = statement case) >>>>>>>>>> 1. Prefork MPM: 2 listen statements: 42% throughput = improvement >>>>>>>>>> 2. Worker MPM: 2 listen statements: 7% throughput = improvement >>>>>>>>>>=20 >>>>>>>>>> In all the above testing cases, significant response time = reductions are observed, even with throughput improvements. >>>>>>>>>>=20 >>>>>>>>>> Please let me know your feedback and comments. >>>>>>>>>>=20 >>>>>>>>>> Thanks, >>>>>>>>>> Yingqi >>>>>>>>>> Software and workloads used in performance tests may have = been optimized for performance only on Intel microprocessors. = Performance tests, such as SYSmark and MobileMark, are measured using = specific computer systems, components, software, operations and = functions. Any change to any of those factors may cause the results to = vary. You should consult other information and performance tests to = assist you in fully evaluating your contemplated purchases, including = the performance of that product when combined with other products. >>>>>>>>>>=20 >>>>>>>>>> From: Lu, Yingqi [mailto:yingqi.lu@intel.com] >>>>>>>>>> Sent: Monday, March 17, 2014 1:41 PM >>>>>>>>>> To: dev@httpd.apache.org >>>>>>>>>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch=20= >>>>>>>>>> with SO_REUSEPORT support >>>>>>>>>>=20 >>>>>>>>>> Dear all, >>>>>>>>>>=20 >>>>>>>>>> Based on the feedback we received, we modified this patch. = Here is the most recent version. We also modified the Bugzilla = database(Bugzilla# 55897 for SO_REUSEPORT patch; Bugzilla# 56279 for = bucket patch). >>>>>>>>>>=20 >>>>>>>>>> Below are the changes we made into this new version: >>>>>>>>>>=20 >>>>>>>>>> According to Yann Ylavic and other people's comments, we = separate the original patch between with and without SO_REUSEPORT into = two separated patches. The SO_REUSEPORT patch does not change the = original listen sockets, it just duplicate the original one into = multiple ones. Since the listen sockets are identical, there is no need = to change the idle_server_maintenance function. The bucket patch = (without SO_REUSEPORT), on the other hand, it breaks down the original = listen record (if there are multiple listen socks) to multiple listen = record linked lists. In this case, idle_server_maintenance is = implemented at bucket level to address the situation that imbalanced = traffic occurs among different listen sockets/children buckets. In the = bucket patch, the polling in the child process is removed since each = child only listens to 1 sock. >>>>>>>>>>=20 >>>>>>>>>> According to Arkadiusz Miskiewicz's comment, we make the = "detection of SO_REUSEPORT" at run time. >>>>>>>>>>=20 >>>>>>>>>> According to Jeff Trawick's comments, 1. We generate the=20 >>>>>>>>>> patches against the httpd trunk. >>>>>>>>>> 2. We tested the current patches and they do not impact event = and worker mpms. If current patches can be accepted, we would be happy = to extend them to other Linux based mpms. There are not much code = changes, but require some time to setup the workload to test. >>>>>>>>>> 3. We removed unnecessary comments and changed APLOGNO(). We = also changed some of the parameter/variable/function names to better = represent their meanings. >>>>>>>>>> 4. There should be no build-in limitations for SO_REUSEPORT = patch. For bucket patch, the only thing is the number of children bucket = only scales to MAX_SPAWN_RATE. If there are more than 32 (current = default MAX_SPQN_RATE) listen statements specified in the httpd.conf, = the number of buckets will be fixed to 32. The reason for this is = because that we implement the idle_server_maintenance at bucket level, = each bucket's own max_spawn_rate is set to MAX_SPAWN_RATE/num_buckets. >>>>>>>>>>=20 >>>>>>>>>> Again, thanks very much for all the comments and feedback. = Please let us know if there are more changes we need to complete to make = them accepted. >>>>>>>>>>=20 >>>>>>>>>> Thanks, >>>>>>>>>> Yingqi Lu >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> From: Lu, Yingqi >>>>>>>>>> Sent: Tuesday, March 04, 2014 10:43 AM >>>>>>>>>> To: dev@httpd.apache.org >>>>>>>>>> Subject: RE: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch=20= >>>>>>>>>> with SO_REUSEPORT support >>>>>>>>>>=20 >>>>>>>>>> Hi Jeff, >>>>>>>>>>=20 >>>>>>>>>> Thanks very much for your time reviewing the patch! We will = modify the patch according to your comments and repost it here. >>>>>>>>>>=20 >>>>>>>>>> Thanks, >>>>>>>>>> Yingqi >>>>>>>>>>=20 >>>>>>>>>> From: Jeff Trawick [mailto:trawick@gmail.com] >>>>>>>>>> Sent: Tuesday, March 04, 2014 10:08 AM >>>>>>>>>> To: Apache HTTP Server Development List >>>>>>>>>> Subject: Re: FW: [PATCH ASF bugzilla# 55897]prefork_mpm patch=20= >>>>>>>>>> with SO_REUSEPORT support >>>>>>>>>>=20 >>>>>>>>>> On Tue, Mar 4, 2014 at 10:35 AM, Lu, Yingqi = wrote: >>>>>>>>>> Hi All, >>>>>>>>>>=20 >>>>>>>>>> I just want to ping again on this patch to gather your = feedback and comments. Please refer to the messages below for patch = details. >>>>>>>>>>=20 >>>>>>>>>> If you need any additional information/supporting data, = please let us know as well. >>>>>>>>>>=20 >>>>>>>>>> Yeah, it has been on my todo list, but I don't have time to=20= >>>>>>>>>> give an in depth review at the moment. Here are a few = questions/comments. >>>>>>>>>> (And you'll have to deal with the fact that it is=20 >>>>>>>>>> unnecessarily tedious for me to evaluate higher-level=20 >>>>>>>>>> considerations if there are a lot of distractions, such as = the=20 >>>>>>>>>> code comments below ;) But others are of course free to chime >>>>>>>>>> in.) >>>>>>>>>>=20 >>>>>>>>>> The patch should be against httpd trunk. It probably won't = take much time for you to create that patch and confirm basic operation. >>>>>>>>>>=20 >>>>>>>>>> What is the impact to other MPMs, even if they shouldn't use = or don't have the necessary code to use SO_REUSEPORT at this time? >>>>>>>>>>=20 >>>>>>>>>> Have you tried the event MPM? >>>>>>>>>>=20 >>>>>>>>>> Is there a way for the admin to choose this behavior? Most = won't care, but everyone's behavior is changed AFAICT. >>>>>>>>>>=20 >>>>>>>>>> Are there built-in limitations in this patch that we should = be aware of? E.g., the free slot/spawn rate changes suggest to me that = there can't be more than 1025 children??? >>>>>>>>>>=20 >>>>>>>>>> We should assume for now that there's no reason this couldn't = be committed to trunk after review/rework, so make sure it is as close = as you can get it to what you think is the final form. >>>>>>>>>>=20 >>>>>>>>>> For the configure-time check for 3.9 kernel: I think we'd = also=20 >>>>>>>>>> use AC_TRY_COMPILE at configure time to confirm that the=20 >>>>>>>>>> SO_REUSEPORT definition is available, and not enable it if = the=20 >>>>>>>>>> system includes doesn't define it. (Does that cause a = problem=20 >>>>>>>>>> for any significant number of people?) >>>>>>>>>>=20 >>>>>>>>>> Don't mention the patch in the patch ;) (e.g., "This function=20= >>>>>>>>>> is added for the patch.") >>>>>>>>>>=20 >>>>>>>>>> Incomplete comments on style/syntax issues: >>>>>>>>>>=20 >>>>>>>>>> * mixing declarations and statements (e.g., "duplr->next =3D = 0;=20 >>>>>>>>>> apr_socket_t *temps;") isn't supported by all compilers and = is=20 >>>>>>>>>> distracting when reviewing >>>>>>>>>> * suitable identifier names (e.g., fix global variable "flag"=20= >>>>>>>>>> and whatever else isn't appropriate; = "ap_post_config_listeners" >>>>>>>>>> should be renamed to indicate what it does >>>>>>>>>> * APLOGNO(99999) and comments about fixing it: Instead put = "APLOGNO()"=20 >>>>>>>>>> and don't add reminders in comments >>>>>>>>>> * this doesn't seem portable: "int = free_slots[MAX_SPAWN_RATE/num_buckets];" >>>>>>>>>> and so on >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> Thanks, >>>>>>>>>> Yingqi >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> From: Lu, Yingqi >>>>>>>>>> Sent: Friday, January 24, 2014 3:26 PM >>>>>>>>>> To: dev@httpd.apache.org >>>>>>>>>> Subject: [PATCH ASF bugzilla# 55897]prefork_mpm patch with=20 >>>>>>>>>> SO_REUSEPORT support >>>>>>>>>>=20 >>>>>>>>>> Dear All, >>>>>>>>>>=20 >>>>>>>>>> Our analysis of Apache httpd 2.4.7 prefork mpm, on 32 and 64 = thread Intel Xeon 2600 series systems, using an open source three tier = social networking web server workload, revealed performance scaling = issues. In current software single listen statement (listen 80) = provides better scalability due to un-serialized accept. However, when = system is under very high load, this can lead to big number of child = processes stuck in D state. On the other hand, the serialized accept = approach cannot scale with the high load either. In our analysis, a = 32-thread system, with 2 listen statements specified, could scale to = just 70% utilization, and a 64-thread system, with signal listen = statement specified (listen 80, 4 network interfaces), could scale to = only 60% utilization. >>>>>>>>>>=20 >>>>>>>>>> Based on those findings, we created a prototype patch for = prefork mpm which extends performance and thread utilization. In Linux = kernel newer than 3.9, SO_REUSEPORT is enabled. This feature allows = multiple sockets listen to the same IP:port and automatically round = robins connections. We use this feature to create multiple duplicated = listener records of the original one and partition the child processes = into buckets. Each bucket listens to 1 IP:port. In case of old kernel = which does not have the SO_REUSEPORT enabled, we modified the "multiple = listen statement case" by creating 1 listen record for each listen = statement and partitioning the child processes into different buckets. = Each bucket listens to 1 IP:port. >>>>>>>>>>=20 >>>>>>>>>> Quick tests of the patch, running the same workload, = demonstrated a 22% throughput increase with 32-threads system and 2 = listen statements (Linux kernel 3.10.4). With the older kernel (Linux = Kernel 3.8.8, without SO_REUSEPORT), 10% performance gain was measured. = With single listen statement (listen 80) configuration, we observed over = 2X performance improvements on modern dual socket Intel platforms (Linux = Kernel 3.10.4). We also observed big reduction in response time, in = addition to the throughput improvement gained in our tests 1. >>>>>>>>>>=20 >>>>>>>>>> Following the feedback from the bugzilla website where we = originally submitted the patch, we removed the dependency of APR change = to simplify the patch testing process. Thanks Jeff Trawick for his good = suggestion! We are also actively working on extending the patch to = worker and event MPMs, as a next step. Meanwhile, we would like to = gather comments from all of you on the current prefork patch. Please = take some time test it and let us know how it works in your environment. >>>>>>>>>>=20 >>>>>>>>>> This is our first patch to the Apache community. Please help = us review it and let us know if there is anything we might revise to = improve it. Your feedback is very much appreciated. >>>>>>>>>>=20 >>>>>>>>>> Configuration: >>>>>>>>>> >>>>>>>>>> ListenBacklog 105384 >>>>>>>>>> ServerLimit 105000 >>>>>>>>>> MaxClients 1024 >>>>>>>>>> MaxRequestsPerChild 0 >>>>>>>>>> StartServers 64 >>>>>>>>>> MinSpareServers 8 >>>>>>>>>> MaxSpareServers 16 >>>>>>>>>> >>>>>>>>>>=20 >>>>>>>>>> 1. Software and workloads used in performance tests may have = been optimized for performance only on Intel microprocessors. = Performance tests, such as SYSmark and MobileMark, are measured using = specific computer systems, components, software, operations and = functions. Any change to any of those factors may cause the results to = vary. You should consult other information and performance tests to = assist you in fully evaluating your contemplated purchases, including = the performance of that product when combined with other products. >>>>>>>>>>=20 >>>>>>>>>> Thanks, >>>>>>>>>> Yingqi >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> -- >>>>>>>>>> Born in Roswell... married an alien... >>>>>>>>>> http://emptyhammock.com/ >>>>>>>>>> http://edjective.org/ >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> -- >>>>>>>>>> Born in Roswell... married an alien... >>>>>>>>>> http://emptyhammock.com/ >>>>>>>>>> http://edjective.org/ >>>>>>>>>>=20 >>>>>>>>>> >>>>>>>>=20 >>>>>>>> >>>>>=20 >>>> >>>=20 >>=20 >=20 >