logging-log4j-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remko Popma <remko.po...@gmail.com>
Subject Re: [jira] [Commented] (LOG4J2-1297) Document "gc-free" configuration and performance
Date Mon, 18 Apr 2016 22:48:14 GMT
Yes, in that case the threadpool doesn't outlive the application so there's no chance of memory
leaks. 

Not sure if Tomcat also can be configured to have separate threadpools per web app and to
refresh the thread pool on web app undeploy/redeploy. 

Sent from my iPhone

> On 2016/04/19, at 3:27, Matt Sicker <boards@gmail.com> wrote:
> 
> But would you say it's safe to use ThreadLocal when you don't undeploy or redeploy web
apps? Like if you always restart Tomcat when you redeploy, would it be safe to use ThreadLocal
in such a scenario?
> 
>> On 18 April 2016 at 09:49, Remko Popma <remko.popma@gmail.com> wrote:
>> The problem is that values stored in ThreadLocals may live longer than the web application
since the threads that hold these values are in a pool and the threads live on after the web
application is undeployed. If the value is a JDK class this is not a problem.
>> 
>> If the value is a class that is defined in your webapp (or a library of the web app)
then holding on the the value means the class cannot be garbage-collected. And voila, we have
a memory leak...
>> 
>> Garbage-free logging makes liberal use of ThreadLocals with non-JDK values. If the
web container separates the thread pools between applications and restarts the thread pool
with the web application, the above memory leak does not occur. (I believe this is what WebSphere
does.)
>> 
>> Eventually we may be able to address this with LOG4J2-1116.
>> 
>> 
>> 
>>> On Mon, Apr 18, 2016 at 11:04 PM, Matt Sicker <boards@gmail.com> wrote:
>>> So is this only bad to use when you have multiple wars on a single Tomcat, or
is it bad for when you try to undeploy, or is there another issue here?
>>> 
>>>> On 18 April 2016 at 09:01, Remko Popma <remko.popma@gmail.com> wrote:
>>>> Yes but if this is wrong because there is no threadpool or so, then you can
use System property "log4j2.is.webapp" to tell Log4j that it should consider your app a non-webapp.
>>>> 
>>>>> On Mon, Apr 18, 2016 at 10:59 PM, Matt Sicker <boards@gmail.com>
wrote:
>>>>> Would an embedded Tomcat/Jetty a la Spring Boot really be considered
a web application?
>>>>> 
>>>>>> On 18 April 2016 at 08:56, Remko Popma <remko.popma@gmail.com>
wrote:
>>>>>> When I finish performance testing I will add graphs to the Performance
section of the gc-free manual page so that users can make an informed decision. (The below
one is an example.)
>>>>>> 
>>>>>> So far, it seems that even with the garbage-free logging enabled,
synchronous logging throughput (the green line) is usually slightly faster than it was with
2.5 (the purple line). So I think making garbage-free the default is acceptable. Also, users
who care about logging throughput will be logging asynchronously, not synchronously: this
scales much better with the number of application threads.
>>>>>> 
>>>>>> 
>>>>>> <ThroughputSyncLinux.png>
>>>>>> ​
>>>>>> 
>>>>>>> On Mon, Apr 18, 2016 at 9:57 PM, Mikael Ståldal <mikael.staldal@magine.com>
wrote:
>>>>>>> Is it wise to enable by default given that "multi-threaded applications
that use synchronous logging may see worse performance"?
>>>>>>> 
>>>>>>>> On Mon, Apr 18, 2016 at 2:22 PM, Remko Popma <remko.popma@gmail.com>
wrote:
>>>>>>>> Enabled, except for web apps; for those log4j will be low
garbage. 
>>>>>>>> 
>>>>>>>> See the first paragraph of the Configuration section in the
manual page. 
>>>>>>>> 
>>>>>>>> Please let me know if this is not clear. (The fact that you
asked means this probably needs improvement...)
>>>>>>>> 
>>>>>>>> Sent from my iPhone
>>>>>>>> 
>>>>>>>>> On 2016/04/18, at 20:18, Mikael Ståldal <mikael.staldal@magine.com>
wrote:
>>>>>>>>> 
>>>>>>>>> Will garbage-free logging be enabled or disabled by default?
>>>>>>>>> 
>>>>>>>>>> On Mon, Apr 18, 2016 at 2:58 AM, Remko Popma (JIRA)
<jira@apache.org> wrote:
>>>>>>>>>> 
>>>>>>>>>>     [ https://issues.apache.org/jira/browse/LOG4J2-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245018#comment-15245018
]
>>>>>>>>>> 
>>>>>>>>>> Remko Popma commented on LOG4J2-1297:
>>>>>>>>>> -------------------------------------
>>>>>>>>>> 
>>>>>>>>>> TODO allocate many temporary objects -> allocate
temporary objects
>>>>>>>>>> 
>>>>>>>>>> TODO multi-threaded applications that use synchronous
logging may see worse performance: the lock that was previously only around the IO operation
is widened to include the text formatting and conversion to bytes. -> _synchronous_ logging
performance may be worse for multi-threaded applications in this mode due to synchronization
on the shared buffer. If your application is multi-threaded and logging performance is important,
consider using Async Loggers.
>>>>>>>>>> 
>>>>>>>>>> TODO Caution: Only dates in the predefined formats
are garbage-free -> Caution: Only the predefined date formats are garbage-free
>>>>>>>>>> 
>>>>>>>>>> TODO "(Note that Log4j may call toString() on message
and parameter objects when garbage-free logging is disabled because system property log4j2.enable.threadlocals
is set to "false".)" -> Log4j may call toString() on message and parameter objects when
garbage-free logging is disabled (when system property log4j2.enable.threadlocals is set to
"false".)
>>>>>>>>>> 
>>>>>>>>>> TODO "made an effort to make logging code garbage-free"
-> "made an effort to make logging garbage-free"
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> > Document "gc-free" configuration and performance
>>>>>>>>>> > ------------------------------------------------
>>>>>>>>>> >
>>>>>>>>>> >                 Key: LOG4J2-1297
>>>>>>>>>> >                 URL: https://issues.apache.org/jira/browse/LOG4J2-1297
>>>>>>>>>> >             Project: Log4j 2
>>>>>>>>>> >          Issue Type: New Feature
>>>>>>>>>> >          Components: Documentation
>>>>>>>>>> >    Affects Versions: 2.5
>>>>>>>>>> >            Reporter: Remko Popma
>>>>>>>>>> >            Assignee: Remko Popma
>>>>>>>>>> >             Fix For: 2.6
>>>>>>>>>> >
>>>>>>>>>> >         Attachments: log4j-2.5-FlightRecording.png,
log4j-2.6-FlightRecording.png
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Update the site with a description of which
configurations are GC-free (i.e., that don't create temporary objects in steady running state).
>>>>>>>>>> > Currently that means
>>>>>>>>>> > * Loggers are all asynchronous (Log4jContextSelector
is set to org.apache.logging.log4j.core.async.AsyncLoggerContextSelector).
>>>>>>>>>> > * The configuration does not contain a <Properties>
section.
>>>>>>>>>> > * The "steady-state" appenders are either RandomAccessFile
or RollingRandomAccessFile Appenders (logging to any other appender will cause temporary objects
to be created - including ConsoleAppender).
>>>>>>>>>> > * The Layout is a PatternLayout that uses one
of the pre-defined date formats, does not have any regular expression replacements, and does
not have lookups (TODO: may need to restrict this further).
>>>>>>>>>> > * The thread name is cached (this is the [default|https://issues.apache.org/jira/browse/LOG4J2-467]).
Running with -DAsyncLogger.ThreadNameStrategy=UNCACHED will create garbage.
>>>>>>>>>> > * In user code, when logging a parameterized
message, the number of parameters is no more than ... (TBD pending discussion in LOG4J2-1278).
>>>>>>>>>> > * In user code, when logging a parameterized
message, parameters of primitive type are boxed in a reused StringBuilder (Log4j provides
a utility to make this relatively painless).
>>>>>>>>>> > Even with the above restrictions, Log4j may
occasionally create garbage:
>>>>>>>>>> > * Initially StringBuilders are presized to 128
characters. They may grow for larger messages (contributing to garbage in Old Gen). If  the
StringBuilder grows beyond 512 characters it is trimmed back to 512 characters to prevent
memory leaks from excessively long messages. (TODO: the resizing algorithm is {{size = value.length
* 2 + 2}}, so a better cutoff value is 518.)
>>>>>>>>>> > * Messages containing {{"$\{"}} will be converted
to a String and StrSubstitutor will be used to replace occurences of variables with their
matching values. Multiple temporary objects are created during this process.
>>>>>>>>>> > Furthermore, we need to explain that some of
this functionality depends on ThreadLocals and so is disabled by default in web applications
to prevent memory leaks. The page should also explain how to manually switch off the use of
ThreadLocals.
>>>>>>>>>> > Finally, the page should show a performance
test comparison similar to the [performance section|http://logging.apache.org/log4j/2.x/manual/async.html#Performance]
on the Async Loggers page. I'm thinking a comparison between Logback, Log4j-1, Log4j-2.0,
Log4j-2.6 "classic" and Log4j-2.6 "gc-free" would be ideal.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> This message was sent by Atlassian JIRA
>>>>>>>>>> (v6.3.4#6332)
>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
>>>>>>>>>> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>>  
>>>>>>>>> 
>>>>>>>>> Mikael Ståldal
>>>>>>>>> Senior software developer 
>>>>>>>>> 
>>>>>>>>> Magine TV
>>>>>>>>> mikael.staldal@magine.com    
>>>>>>>>> Grev Turegatan 3  | 114 46 Stockholm, Sweden  |   www.magine.com

>>>>>>>>> 
>>>>>>>>> Privileged and/or Confidential Information may be contained
in this message. If you are not the addressee indicated in this message
>>>>>>>>> (or responsible for delivery of the message to such a
person), you may not copy or deliver this message to anyone. In such case, 
>>>>>>>>> you should destroy this message and kindly notify the
sender by reply email.   
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>>  
>>>>>>> 
>>>>>>> Mikael Ståldal
>>>>>>> Senior software developer 
>>>>>>> 
>>>>>>> Magine TV
>>>>>>> mikael.staldal@magine.com    
>>>>>>> Grev Turegatan 3  | 114 46 Stockholm, Sweden  |   www.magine.com

>>>>>>> 
>>>>>>> Privileged and/or Confidential Information may be contained in
this message. If you are not the addressee indicated in this message
>>>>>>> (or responsible for delivery of the message to such a person),
you may not copy or deliver this message to anyone. In such case, 
>>>>>>> you should destroy this message and kindly notify the sender
by reply email.   
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Matt Sicker <boards@gmail.com>
>>> 
>>> 
>>> 
>>> -- 
>>> Matt Sicker <boards@gmail.com>
> 
> 
> 
> -- 
> Matt Sicker <boards@gmail.com>

Mime
View raw message