ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Drazen Zubovic (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMBARI-13740) Corrupt 'spark-log4j-properties' for Spark service
Date Thu, 05 Nov 2015 13:09:28 GMT
Drazen Zubovic created AMBARI-13740:
---------------------------------------

             Summary: Corrupt 'spark-log4j-properties' for Spark service
                 Key: AMBARI-13740
                 URL: https://issues.apache.org/jira/browse/AMBARI-13740
             Project: Ambari
          Issue Type: Bug
          Components: ambari-admin, ambari-client
    Affects Versions: 2.1.0
         Environment: Hortonworks sandbox 2.3.1; HDP-2.3.0.0-2557
            Reporter: Drazen Zubovic
            Priority: Minor


It seems like spark-log4j-properties is getting corrupt and reformatted into one line without
LF. Like the log4j config is field with LF chars masked out. The configuration cannot be changed
into multi line configuration meaningful to Spark. Consequently console prints INFO messages
for each statement. 

The log4j.properties.template configuration looks like this: 

# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

The log4j.properties configuration looks like this: 

# Set everything to be logged to the consolelog4j.rootCategory=WARN, consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.errlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd
HH:mm:ss} %p %c{1}: %m%n# Settings to quiet third party logs that are too verboselog4j.logger.org.eclipse.jetty=WARNlog4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERRORlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFOlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO


This issue leads to the ### INFO messages that log4j prints in Spark console:
[root@sandbox ~]# spark-shell --master yarn-client --driver-memory 512m --executor-memory
512m
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/11/05 12:34:41 INFO SecurityManager: Changing view acls to: root
15/11/05 12:34:41 INFO SecurityManager: Changing modify acls to: root
15/11/05 12:34:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls
disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/11/05 12:34:41 INFO HttpServer: Starting HTTP Server
15/11/05 12:34:41 INFO Server: jetty-8.y.z-SNAPSHOT
15/11/05 12:34:41 INFO AbstractConnector: Started SocketConnector@0.0.0.0:54564
15/11/05 12:34:41 INFO Utils: Successfully started service 'HTTP class server' on port 54564.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.3.1
      /_/

Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_85)
Type in expressions to have them evaluated.
Type :help for more information.
15/11/05 12:34:44 INFO SparkContext: Running Spark version 1.3.1
15/11/05 12:34:44 INFO SecurityManager: Changing view acls to: root
15/11/05 12:34:44 INFO SecurityManager: Changing modify acls to: root
15/11/05 12:34:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls
disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/11/05 12:34:44 INFO Slf4jLogger: Slf4jLogger started
15/11/05 12:34:44 INFO Remoting: Starting remoting
15/11/05 12:34:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@sandbox.hortonworks.com:35020]
15/11/05 12:34:44 INFO Utils: Successfully started service 'sparkDriver' on port 35020.
15/11/05 12:34:44 INFO SparkEnv: Registering MapOutputTracker
15/11/05 12:34:44 INFO SparkEnv: Registering BlockManagerMaster
15/11/05 12:34:44 INFO DiskBlockManager: Created local directory at /tmp/spark-3b14e9fc-73b1-45e7-96f3-f0ee48f3a238/blockmgr-ffd0619a-5fbc-4ccc-bcef-ce8560e03cd7
15/11/05 12:34:44 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/11/05 12:34:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-7648f575-a8e3-4b25-945f-b4ded8631592/httpd-ec5526bb-4de6-42d6-83aa-a1ba0a2ca5dd
15/11/05 12:34:45 INFO HttpServer: Starting HTTP Server
15/11/05 12:34:45 INFO Server: jetty-8.y.z-SNAPSHOT
15/11/05 12:34:45 INFO AbstractConnector: Started SocketConnector@0.0.0.0:58853
15/11/05 12:34:45 INFO Utils: Successfully started service 'HTTP file server' on port 58853.
15/11/05 12:34:45 INFO SparkEnv: Registering OutputCommitCoordinator
1




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message