spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yong Zhang <>
Subject RE: Logging in executors
Date Wed, 13 Apr 2016 17:35:39 GMT
Is the env/dev/ file within your jar file? Is the path matching with
what you specified as env/dev/
If you read the log4j document here:
When you specify the, you have 2 option:
1) the has to be in the jar (or in the classpath). In your case, since
you specify the package path, you need to make sure they are matched in your jar file2) use
like log4j.configuration=file:///tmp/ In this way, you need to make sure
file exists in /tmp folder on ALL of your worker nodes.

Date: Wed, 13 Apr 2016 14:18:24 -0300
Subject: Re: Logging in executors

Thanks for your response Ted. You're right, there was a typo. I changed it, now I'm executing:
bin/spark-submit --master spark://localhost:7077 --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=env/dev/"
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=env/dev/"
The content of this file is:
# Set everything to be logged to the consolelog4j.rootCategory=INFO,{yy/MM/dd
HH:mm:ss} %p %c{1}: %m%n
HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too$$
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL
with Hive

Finally, the code on which I'm using logging in the executor is:
def groupAndCount(keys: DStream[(String, List[String])])(handler: ResultHandler) = {
  val result = keys.reduceByKey((prior, current) => {
    (prior ::: current)
  }).flatMap {
    case (date, keys) =>
      val rs = keys.groupBy(x => x).map(
          obs =>{
            val (d,t) = date.split("@") match {
              case Array(d,t) => (d,t)
            import org.apache.log4j.Logger
            import scala.collection.JavaConverters._
            val logger: Logger = Logger.getRootLogger
  "Metric retrieved $d")
            Metric("PV", d, obs._1, t, obs._2.size)

  result.foreachRDD((rdd: RDD[Metric], time: Time) => {
    handler(rdd, time)

Originally the import and logger object was outside the map function. I'm also using the root
logger just to see if it's working, but nothing gets logged. I've checked that the property
is set correctly on the executor side through println(System.getProperty("log4j.configuration"))
and is OK, but still not working.
Thanks again,-carlos. 		 	   		  
View raw message