From reviews-return-673977-archive-asf-public=cust-asf.ponee.io@spark.apache.org Wed Jul 18 10:07:31 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DB7FC180636 for ; Wed, 18 Jul 2018 10:07:30 +0200 (CEST) Received: (qmail 84590 invoked by uid 500); 18 Jul 2018 08:07:29 -0000 Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@spark.apache.org Received: (qmail 84579 invoked by uid 99); 18 Jul 2018 08:07:29 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2018 08:07:29 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 1607EDFA6D; Wed, 18 Jul 2018 08:07:29 +0000 (UTC) From: HyukjinKwon To: reviews@spark.apache.org Reply-To: reviews@spark.apache.org References: In-Reply-To: Subject: [GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output Content-Type: text/plain Message-Id: <20180718080729.1607EDFA6D@git1-us-west.apache.org> Date: Wed, 18 Jul 2018 08:07:29 +0000 (UTC) Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 OK, but you can set sc.setLogLevel in the shell. For instance, if I run `spark.range(10).show()` with INFO, I got something like this: ``` scala> 18/07/18 07:58:47 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) spark.r18/07/18 07:58:48 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) ange(10)18/07/18 07:58:49 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) .show()18/07/18 07:58:50 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:51 INFO internal.SharedState: loading hive config file: file:/home/spark/spark/conf/hive-site.xml 18/07/18 07:58:51 INFO internal.SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/spark/spark/spark-warehouse'). 18/07/18 07:58:51 INFO internal.SharedState: Warehouse path is 'file:/home/spark/spark/spark-warehouse'. 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7a68818c{/SQL,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5f745970{/SQL/json,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2afd8972{/SQL/execution,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5784f6b9{/SQL/execution/json,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6ccf06f1{/static/sql,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:52 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 18/07/18 07:58:52 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 254.142542 ms 18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 65.397101 ms 18/07/18 07:58:53 INFO spark.SparkContext: Starting job: show at :24 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Got job 0 (show at :24) with 1 output partitions 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (show at :24) 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Parents of final stage: List() 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Missing parents: List() 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at show at :24), which has no missing parents 18/07/18 07:58:53 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 8.1 KB, free 408.9 MB) 18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.0 KB, free 408.9 MB) 18/07/18 07:58:53 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ctr-e138-1518143905142-411342-01-000002.hwx.site:41073 (size: 4.0 KB, free: 408.9 MB) 18/07/18 07:58:53 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1039 18/07/18 07:58:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at show at :24) (first 15 tasks are for partitions Vector(0)) 18/07/18 07:58:54 INFO cluster.YarnScheduler: Adding task set 0.0 with 1 tasks 18/07/18 07:58:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, ctr-e138-1518143905142-411342-01-000008.hwx.site, executor 2, partition 0, PROCESS_LOCAL, 7864 bytes) [Stage 0:> (0 + 1) / 1]18/07/18 07:58:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ctr-e138-1518143905142-411342-01-000008.hwx.site:37671 (size: 4.0 KB, free: 366.3 MB) 18/07/18 07:58:54 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1115 ms on ctr-e138-1518143905142-411342-01-000008.hwx.site (executor 2) (1/1) 18/07/18 07:58:55 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 18/07/18 07:58:55 INFO scheduler.DAGScheduler: ResultStage 0 (show at :24) finished in 1.609 s 18/07/18 07:58:55 INFO scheduler.DAGScheduler: Job 0 finished: show at :24, took 1.673348 s 18/07/18 07:58:55 INFO spark.SparkContext: Starting job: show at :24 18/07/18 07:58:55 INFO scheduler.DAGScheduler: Got job 1 (show at :24) with 1 output partitions 18/07/18 07:58:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (show at :24) 18/07/18 07:58:55 INFO scheduler.DAGScheduler: Parents of final stage: List() 18/07/18 07:58:55 INFO scheduler.DAGScheduler: Missing parents: List() 18/07/18 07:58:55 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[4] at show at :24), which has no missing parents 18/07/18 07:58:55 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 8.1 KB, free 408.9 MB) 18/07/18 07:58:55 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.0 KB, free 408.9 MB) 18/07/18 07:58:55 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on ctr-e138-1518143905142-411342-01-000002.hwx.site:41073 (size: 4.0 KB, free: 408.9 MB) 18/07/18 07:58:55 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039 18/07/18 07:58:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[4] at show at :24) (first 15 tasks are for partitions Vector(1)) 18/07/18 07:58:55 INFO cluster.YarnScheduler: Adding task set 1.0 with 1 tasks 18/07/18 07:58:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, ctr-e138-1518143905142-411342-01-000007.hwx.site, executor 1, partition 1, PROCESS_LOCAL, 7864 bytes) 18/07/18 07:58:55 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on ctr-e138-1518143905142-411342-01-000007.hwx.site:33073 (size: 4.0 KB, free: 366.3 MB) 18/07/18 07:58:55 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) [Stage 1:> (0 + 1) / 1]18/07/18 07:58:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 948 ms on ctr-e138-1518143905142-411342-01-000007.hwx.site (executor 1) (1/1) 18/07/18 07:58:56 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 18/07/18 07:58:56 INFO scheduler.DAGScheduler: ResultStage 1 (show at :24) finished in 0.956 s 18/07/18 07:58:56 INFO scheduler.DAGScheduler: Job 1 finished: show at :24, took 0.959303 s +---+ | id| +---+ | 0| | 1| | 2| | 3| | 4| | 5| | 6| | 7| | 8| | 9| +---+ ``` Considering it's info level, I was thinking it's not super noisy. I am okay with disabling though. I got your point. Let me defer this to @vanzin and @jerryshao. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org For additional commands, e-mail: reviews-help@spark.apache.org