Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5B6CB200ACA for ; Thu, 19 May 2016 01:34:56 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 59DE4160A15; Wed, 18 May 2016 23:34:56 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2C70F160A00 for ; Thu, 19 May 2016 01:34:55 +0200 (CEST) Received: (qmail 86040 invoked by uid 500); 18 May 2016 23:34:54 -0000 Mailing-List: contact log4j-dev-help@logging.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Help: List-Post: List-Id: "Log4J Developers List" Reply-To: "Log4J Developers List" Delivered-To: mailing list log4j-dev@logging.apache.org Received: (qmail 86027 invoked by uid 99); 18 May 2016 23:34:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2016 23:34:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D03FD1A5453 for ; Wed, 18 May 2016 23:34:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id PO7WtOCvfnFd for ; Wed, 18 May 2016 23:34:51 +0000 (UTC) Received: from mail-io0-f187.google.com (mail-io0-f187.google.com [209.85.223.187]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id AC0CE5F1F7 for ; Wed, 18 May 2016 23:34:50 +0000 (UTC) Received: by mail-io0-f187.google.com with SMTP id s97so15597232ioi.1 for ; Wed, 18 May 2016 16:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version; bh=vUUTvRnqx9BpHuN9hm5824SROF0C3LOh0ergLjcfp5Y=; b=TONK5sUGWLG/D4OIUG7STVXLnn9KBEcQnXZrp9FcH6wm2dSWas+pDUbCVB6R6MjuMk +IxaTHh9/1Y5jB3h68TKsFm8sbCm3c2UK920gZn8bdzbmP+xSQa//kqS6oQJ4vOm3WK3 dqcGdBsH4Qlu9g2eZyvUYLeCgDslBL+sZIyBnIUURSRgh98dvXJECf+Z+QI6Zo3hpyrX v8y7kMzPT1Fb/KbxC6Ui1SZH5ubMArRuOWvxVMZ33H0u09y6ZJ2jirRM7TLRoz7rn1Qs pVYgHnuwjvGvGPIYqb4q3Vyob8bdtyOzMvQy8fIyz17s+mj9+QFHm+3cvbtG1r3hNckg PCrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:message-id:in-reply-to :references:subject:mime-version; bh=vUUTvRnqx9BpHuN9hm5824SROF0C3LOh0ergLjcfp5Y=; b=ma8jqJZMEemjbNj+CSoC9eXXBByc+vWujEu+rMr7L8zkp2+MHym+BdXIPuhnJ4YWr7 90K12BEPLrSX125SsEJQi+pHheJjEli2fU3UW2Qxhu6O0SyKNqZ+lpbXPzdH9K1x4Otw B9yoZtDih83LbXYYZ1aQ2HY3JYBc/R+layIcyuUPqt8krJvjNpr9KXrlczDQ93b8sxPp vcEsOivvqkt74ctXk/2tw9rFn1pHV2Dx+K3u9O3XwCS9/epAme70VaygJ0yFvdnglnOH +BscpI/U5VAnzDdhBXzvGRMUPGi3mwQ/XTlXhu9yObv4HzK7RpLgu0KxYNuYFZQRA9B+ HBmA== X-Gm-Message-State: AOPr4FWfm20NizLadfAo+dBOSCb2NLsC7lA1rvD33/l3OAtRgbHEIE5VEe+mQ2L39o7TvTkstca1 X-Received: by 10.140.98.247 with SMTP id o110mr1344679qge.18.1463614483683; Wed, 18 May 2016 16:34:43 -0700 (PDT) X-Google-Already-Archived: Yes X-Google-Already-Archived-Group-Id: 96d53ff93e X-Google-Doc-Id: ae3a75d7d2eb X-Google-Thread-Id: 92579f8d1aa5a501 X-Google-Message-Url: http://groups.google.com/group/mechanical-sympathy/msg/ae3a75d7d2eb X-Google-Thread-Url: http://groups.google.com/group/mechanical-sympathy/t/92579f8d1aa5a501 X-Google-Web-Client: true Date: Wed, 18 May 2016 16:34:42 -0700 (PDT) From: Remko Popma To: mechanical-sympathy Cc: _@belliottsmith.com, Log4J Developers List Message-Id: <352369d7-b92f-4e8a-81c5-2a0e5b29ff05@googlegroups.com> In-Reply-To: References: <10013c64-b50f-43dd-a430-dd1ec637769c@googlegroups.com> Subject: Re: Garbage-free Log4j docs preview MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_90_1447838141.1463614482918" X-Google-Token: EJL487kFZFGjb7hPY_s0 X-Google-IP: 115.179.87.3 archived-at: Wed, 18 May 2016 23:34:56 -0000 ------=_Part_90_1447838141.1463614482918 Content-Type: multipart/alternative; boundary="----=_Part_91_699044553.1463614482919" ------=_Part_91_699044553.1463614482919 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Benedict, thanks for your feedback! With regards to filtering, global filters are already considered before the event is enqueued. Filters configured on the Logger and AppenderRef are applied prior to enqueueing with mixed Sync/Async Loggers but when all loggers are async these filters are applied by the background thread. We can probably improve this, thanks for the suggestion! Your suggestion to run performance tests under lower loads is reasonable and we will look at this for a future release. I did experiment with this a little a while back and did see larger response time pauses. Perhaps others with more experience can chime in. My understanding of how the Disruptor works is that enqueueing the event is lock-free, so applications threads doing the logging should not experience any context switches or suffer latency from Futex calls caused by logging (regardless of the workload). The background thread is another story. Under moderate to low workloads the background thread may (depending on the wait policy) fall asleep and experience delays before waking up when work arrives. However, if there are enough cores to serve all threads I don't see how such background thread delays can impact (cause response time pauses for) the application that is doing the logging. Please correct me if my understanding is incorrect. My thinking is that using async loggers is good for reactive applications that need to respond quickly to external events. It is especially useful if the application needs to deal with occasional bursts of work (and accompanied bursts of logging). This is where async loggers can deliver value even if the normal workload is low. Remko On Wednesday, May 18, 2016 at 2:56:18 AM UTC+9, Benedict Elliott Smith wrote: > > Regrettably it seems impossible (or at least very annoying) to send to > both lists simultaneously... > > On 17 May 2016 at 18:55, Benedict Elliott Smith > wrote: > >> Could I suggest that you run tests for latency and throughput effects of >> using this in systems with only moderate-to-low numbers of logging calls? >> (i.e. the vast majority of systems!) >> > >> It's very likely that in such systems messages will not reach a velocity >> to keep the Dispatcher running, and so logging calls may often (or always) >> involve a Futex call - which is not a trivial cost. There will almost >> certainly be systems out there with anti-ideal characteristics - logging >> just often enough that these costs materially impact throughput, but not >> often enough that they suddenly disappear. >> >> Even though a majority of systems *do not need this*, because it "async" >> and the new hotness, and there are no advertised downsides, everyone will >> try to use it. It's up to those who know better to make sure these people >> are informed it isn't a free lunch. Making sure all of the caveats are >> reported on the advertising page would be a great start IMO. >> >> Might I also separately suggest you consider filtering events prior to >> placing them on the queue for processing by the dispatcher? I've only >> briefly glanced at the code, but it seems not to be the case currently. >> >>> >>> On 17 May 2016 at 18:33, Martin Thompson >>> > wrote: >>> >>>> Hi Remko, >>>> >>>> I'd just like to say that it is a great service to the community as a >>>> whole that someone is seriously looking at improving logging. >>>> >>>> If you keep it up you'll be putting folks like me out of a job :) >>>> >>>> Martin... >>>> >>>> >>>> On 17 May 2016 at 18:13, Remko Popma > >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> First, my thanks to the many people who gave helpful advice and >>>>> feedback on how to measure Log4j response time on this list some time ago. >>>>> >>>>> We're about to start the Log4j 2.6 release. >>>>> If anyone is interested, a preview of the garbage-free logging manual >>>>> page is here: >>>>> http://home.apache.org/~rpopma/log4j/2.6/manual/garbagefree.html >>>>> and a preview of the updated performance page is here: >>>>> http://home.apache.org/~rpopma/log4j/2.6/performance.html >>>>> >>>>> Feedback welcome! >>>>> >>>>> Remko >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "mechanical-sympathy" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to mechanical-sympathy+unsubscribe@googlegroups.com >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "mechanical-sympathy" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to mechanical-sympathy+unsubscribe@googlegroups.com >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >> > ------=_Part_91_699044553.1463614482919 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Benedict, thanks for your feedback!

Wit= h regards to filtering, global filters are already considered before the ev= ent is enqueued. Filters configured on the Logger and AppenderRef are appli= ed prior to enqueueing with mixed Sync/Async Loggers but when al= l loggers are async these filters are applied by the background thread.= We can probably improve this, thanks for the suggestion!

Your suggestion to run performance tests under lower loads is reaso= nable and we will look at this for a future release.
I did experi= ment with this a little a while back and did see larger response time pause= s. Perhaps others with more experience can chime in.

My understanding of how the Disruptor works is that enqueueing the event= is lock-free, so applications threads doing the logging should not experie= nce any context switches or suffer latency from Futex calls caused by loggi= ng (regardless of the workload). The background thread is another story. Un= der moderate to low workloads the background thread may (depending on the w= ait policy) fall asleep and experience delays before waking up when work ar= rives. However, if there are enough cores to serve all threads I don't = see how such background thread delays can impact (cause response time pause= s for) the application that is doing the logging. Please correct me if my u= nderstanding is incorrect.

My thinking is that usi= ng async loggers is good for reactive applications that need to respond qui= ckly to external events. It is especially useful if the application needs t= o deal with occasional bursts of work (and accompanied bursts of logging). = This is where async loggers can deliver value even if the normal workload i= s low.

Remko

On Wednesday, May 18, 2= 016 at 2:56:18 AM UTC+9, Benedict Elliott Smith wrote:
Regrettably it seems impossible (o= r at least very annoying) to send to both lists simultaneously...

<= div class=3D"gmail_quote">On 17 May 2016 at 18:55, Benedict Elliott Smith <= span dir=3D"ltr"><bene...@apache.org> wrote:
Could I s= uggest that you run tests for latency and throughput effects of using this = in systems with only moderate-to-low numbers of logging calls? =C2=A0(i.e. = the vast majority of systems!)
<= /blockquote>
<= div>

It's very likely that in such systems messages will not reach a velo= city to keep the Dispatcher running, and so logging calls may often (or alw= ays) involve a Futex call - which is not a trivial cost.=C2=A0 There will a= lmost certainly be systems out there with anti-ideal characteristics - logg= ing just often enough that these costs materially impact throughput, but no= t often enough that they suddenly disappear. =C2=A0

Even though a major= ity of systems=C2=A0do not need this, because it "async" a= nd the new hotness, and there are no advertised downsides, everyone will tr= y to use it.=C2=A0 It's up to those who know better to make sure these = people are informed it isn't a free lunch.=C2=A0 Making sure all of the= caveats are reported on the advertising page would be a great start IMO.

Might I also separately suggest you consider filtering events prior to p= lacing them on the queue for processing by the dispatcher?=C2=A0 I've o= nly briefly glanced at the code, but it seems not to be the case currently.=
=

On 17 May 2016 at 1= 8:33, Martin Thompson <mjp...@gmail.com> wro= te:
Hi Remko,

I'd just like to say that it is a great service to the communi= ty as a whole that someone is seriously looking at improving logging.
=

If you keep it up you'll be putting folks like me o= ut of a job :)

Marti= n...


On 17 May 2016 at 18:13, Remko Popma <= ;remk= o...@gmail.com> wrote:
Hi all,

First, my thanks to the many people= who gave helpful advice and feedback on how to measure Log4j response time= on this list some time ago.

We're about to st= art the Log4j 2.6 release.
If anyone is interested, a preview of = the garbage-free logging manual page is here:=C2=A0http://home.apache.org/~rpopma/log4j/2.6/manua= l/garbagefree.html
and a preview of the updated performa= nce page is here:=C2=A0http://home.apache.org/~r= popma/log4j/2.6/performance.html

Feedback= welcome!

Remko

--
You received this message because you are subscribed to the Google Groups &= quot;mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to mechanical-sympathy+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/= d/optout.

--
You received this message because you are subscribed to the Google Groups &= quot;mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to mechanical-sympathy+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/= d/optout.



------=_Part_91_699044553.1463614482919-- ------=_Part_90_1447838141.1463614482918 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org For additional commands, e-mail: log4j-dev-help@logging.apache.org ------=_Part_90_1447838141.1463614482918--