maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason van Zyl <ja...@takari.io>
Subject Re: logging during multithreaded builds
Date Thu, 28 May 2015 03:21:00 GMT
I honestly don't think an optional feature relying on an optional dependency belongs in the
core itself.

On May 27, 2015, at 10:34 PM, Igor Fedorenko <igor@ifedorenko.com> wrote:

> There are three semi-related parts to my implementation
> 
> 1. SLF4J MDC management, basically setting and removing project
> attributes in a thread-local map. Truly reliable implementation will
> need to be coded in all Builders. Alternatively, it should be possible
> to use existing lifecycle callbacks to implement mostly reliable lib/ext
> extension.
> 
> 2. Install and configure System out/err and java.util.logging bridges.
> This can be implemented as lib/ext extension using existing callbacks
> 
> 3. Custom logback appenders and configuration that provide "interesting"
> behaviour using SLF4J MDC. These too can be implemented of the core.
> 
> Keeping all this in the core will allow marginally more reliable SLF4J
> MDC management implementation and I also think will make it more likely
> for other developers to help with the implementation. It will also serve
> as a working example for adding support for other logging framework. But
> I agree, this does not have to be implemented in the core so I'll try to
> rework my implementation as lib/ext extension.
> 
> -- 
> Regards,
> Igor
> 
> On Wed, May 27, 2015, at 06:40 PM, Jason van Zyl wrote:
>> 
>> On May 27, 2015, at 3:55 PM, Igor Fedorenko <igor@ifedorenko.com> wrote:
>> 
>>> So I went ahead and implemented these changes, including working (but
>>> not terribly well tested) logback appenders to buffer-and-group project
>>> console log messages and create per-project build.log files. 
>>> 
>> 
>> What changes were required in the core?
>> 
>>> Does anyone see a problem if I check in these appenders in maven core
>>> source tree or you prefer me to keep them elsewhere? 
>>> 
>> 
>> If it's required to alter the distribution to install logback in order to
>> use the appenders then I think elsewhere is more appropriate. As they
>> won't be used by the standard version of Maven so I don't think it
>> belongs in core along with the other modules. Maven shared is probably
>> fine.
>> 
>>> Just to be clear, I do not propose to "hardware" maven to logback and I
>>> do not propose to include logback support in maven distribution. I want
>>> to introduce new maven-ext-logback module, which users will be able to
>>> use to customize their maven distributions very much the same way they
>>> need to do it now to use any of the advanced logging frameworks.
>>> 
>>> -- 
>>> Regards,
>>> Igor
>>> 
>>> On Tue, May 26, 2015, at 03:38 PM, Igor Fedorenko wrote:
>>>> I spent some time looking into this, and I think project-level logging
>>>> will require several semi-related changes.
>>>> 
>>>> * As Ralph pointed out, Maven needs to use SLF4J MDC to associate log
>>>> messages with individual projects being built. This is required to
>>>> enable any project-related logging approach and I plan to submit this
>>>> change in next couple of days.
>>>> 
>>>> * Another problem is use of non-slf4j logging techniques by Maven
>>>> plugins. Direct use of System out/err and java.util.logging are two
>>>> obvious problems and I plan to change maven to "bridge" these to slf4j. 
>>>> 
>>>> * The old log4j 1.x logging and commons-logging need to be bridged to
>>>> slf4j too. I can make this change in maven, but this is not strictly
>>>> necessary because it can be done from lib/ext extension too.
>>>> 
>>>> The rest really depends on whether we can agree on single "advanced"
>>>> logging backend and if we need to support several logging
>>>> configurations. I think we've found three viable project-level logging
>>>> approaches: buffered console output, better logging pattern and my
>>>> original target/build.log idea. Unless somebody really wants to restart
>>>> logback-vs-log4j discussion, I suggest we postpone this decision and
>>>> instead implement project-level logging as out-of-core extensions.
>>>> 
>>>> Does this make sense?
>>>> 
>>>> -- 
>>>> Regards,
>>>> Igor
>>>> 
>>>> On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
>>>>> If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and
>>>>> 2.x - you can include anything stored in the MDC on every line of log
>>>>> output.  Just use %X to include all MDC items or %MDC{key} to include
the
>>>>> specific key.  This would require storing the value(s) at the beginning
>>>>> of every thread.
>>>>> 
>>>>> If you include %t in the pattern than every log line should include the
>>>>> threadId.
>>>>> 
>>>>> If you are saying that the log lines include newlines in them that should
>>>>> still be OK. The only way the lines should be mangled is if each thread
>>>>> somehow has its own instance of the logging framework and they are all
>>>>> configured to write to the same file.
>>>>> 
>>>>> Ralph
>>>>> 
>>>>> 
>>>>> 
>>>>>> On May 25, 2015, at 7:28 AM, Igor Fedorenko <igor@ifedorenko.com>
wrote:
>>>>>> 
>>>>>> Yes, thread-id will help to some degree, but maven uses multiline
log
>>>>>> messages quite often and these will still be mangled and unreadable.
>>>>>> Per-project build log files is the only solution I found to preserve
>>>>>> readable logs. Also, each project build is mostly independent from
the
>>>>>> rest and I find reading self-contain per-project log files much easier.
>>>>>> 
>>>>>> -- 
>>>>>> Regards,
>>>>>> Igor
>>>>>> 
>>>>>> On Mon, May 25, 2015, at 10:21 AM, Sean Busbey wrote:
>>>>>>> In multithreaded builds we could add a thread ID to each output
line.
>>>>>>> That
>>>>>>> would make it easier to read and filter in different files in
post
>>>>>>> processing.
>>>>>>> 
>>>>>>> -- 
>>>>>>> Sean
>>>>>>> On May 25, 2015 6:30 AM, "Igor Fedorenko" <igor@ifedorenko.com>
wrote:
>>>>>>> 
>>>>>>>> I had to troubleshoot a large multithreaded build last week
and that
>>>>>>>> proved to be rather difficult mostly because build log was
a jumble of
>>>>>>>> messages produced by concurrently running threads. It was
not possible
>>>>>>>> to tell which message came from which thread, which made
the build log
>>>>>>>> more or less useless.
>>>>>>>> 
>>>>>>>> What I ended up doing was to write per-module log message
to individual
>>>>>>>> ${project.build.directory}/build.log log files.
>>>>>>>> 
>>>>>>>> That was kinda tricky to implement because log files were
opened very
>>>>>>>> early during module build and were subsequently deleted by
>>>>>>>> maven-clean-plugin (I tried on Linux and OSX, and I assume
the build
>>>>>>>> will fail on Windows). I had to modify maven-clean-plugin
configuration
>>>>>>>> in the project pom.xml to retain ${project.build.directory}/build.log
>>>>>>>> files.
>>>>>>>> 
>>>>>>>> I felt my solution required too my effort and I wonder what
others do to
>>>>>>>> capture logs during multithreaded builds. Can we come up
with a
>>>>>>>> "recommended" way of doing this?
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Igor
>>>>>>>> 
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>> 
>> 
>> Thanks,
>> 
>> Jason
>> 
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder, Takari and Apache Maven
>> http://twitter.com/jvanzyl
>> http://twitter.com/takari_io
>> ---------------------------------------------------------
>> 
>> To do two things at once is to do neither.
>> 
>> -- Publilius Syrus, Roman slave, first century B.C.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
> 

Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder, Takari and Apache Maven
http://twitter.com/jvanzyl
http://twitter.com/takari_io
---------------------------------------------------------

The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that
is, 
the search for a superior moral justification for selfishness.

 -- John Kenneth Galbraith













---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message