spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: 0.9.0 forces log4j usage
Date Fri, 07 Feb 2014 08:05:40 GMT
A config option e.g. could just be to add:

spark.logging.loadDefaultLogger (default true)
If set to true, Spark will try to initialize a log4j logger if none is
detected. Otherwise Spark will not modify logging behavior.

Then users could just set this to false if they have a logging set-up
that conflicts with this.

Maybe there is a nicer fix...

On Fri, Feb 7, 2014 at 12:03 AM, Patrick Wendell <pwendell@gmail.com> wrote:
> Hey Paul,
>
> Thanks for digging this up. I worked on this feature and the intent
> was to give users good default behavior if they didn't include any
> logging configuration on the classpath.
>
> The problem with assuming that CL tooling is going to fix the job is
> that many people link against spark as a library and run their
> application using their own scripts. In this case the first thing
> people see when they run an application that links against Spark was a
> big ugly logging warning.
>
> I'm not super familiar with log4j-over-slf4j, but this behavior of
> returning null for the appenders seems a little weird. What is the use
> case for using this and not just directly use slf4j-log4j12 like Spark
> itself does?
>
> Did you have a more general fix for this in mind? Or was your plan to
> just revert the existing behavior... We might be able to add a
> configuration option to disable this logging default stuff. Or we
> could just rip it out - but I'd like to avoid that if possible.
>
> - Patrick
>
> On Thu, Feb 6, 2014 at 11:41 PM, Paul Brown <prb@mult.ifario.us> wrote:
>> We have a few applications that embed Spark, and in 0.8.0 and 0.8.1, we
>> were able to use slf4j, but 0.9.0 broke that and unintentionally forces
>> direct use of log4j as the logging backend.
>>
>> The issue is here in the org.apache.spark.Logging trait:
>>
>> https://github.com/apache/incubator-spark/blame/master/core/src/main/scala/org/apache/spark/Logging.scala#L107
>>
>> log4j-over-slf4j *always* returns an empty enumeration for appenders to the
>> ROOT logger:
>>
>> https://github.com/qos-ch/slf4j/blob/master/log4j-over-slf4j/src/main/java/org/apache/log4j/Category.java?source=c#L81
>>
>> And this causes an infinite loop and an eventual stack overflow.
>>
>> I'm happy to submit a Jira and a patch, but it would be significant enough
>> reversal of recent changes that it's probably worth discussing before I
>> sink a half hour into it.  My suggestion would be that initialization (or
>> not) should be left to the user with reasonable default behavior supplied
>> by the spark commandline tooling and not forced on applications that
>> incorporate Spark.
>>
>> Thoughts/opinions?
>>
>> -- Paul
>> --
>> prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/

Mime
View raw message