spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HyukjinKwon <...@git.apache.org>
Subject [GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Date Wed, 18 Jul 2018 08:07:29 GMT
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/21784
  
    OK, but you can set sc.setLogLevel in the shell. For instance, if I run `spark.range(10).show()`
with INFO, I got something like this:
    
    ```
    scala> 18/07/18 07:58:47 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    spark.r18/07/18 07:58:48 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    ange(10)18/07/18 07:58:49 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    .show()18/07/18 07:58:50 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    
    18/07/18 07:58:51 INFO internal.SharedState: loading hive config file: file:/home/spark/spark/conf/hive-site.xml
    18/07/18 07:58:51 INFO internal.SharedState: Setting hive.metastore.warehouse.dir ('null')
to the value of spark.sql.warehouse.dir ('file:/home/spark/spark/spark-warehouse').
    18/07/18 07:58:51 INFO internal.SharedState: Warehouse path is 'file:/home/spark/spark/spark-warehouse'.
    18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
to /SQL.
    18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7a68818c{/SQL,null,AVAILABLE,@Spark}
    18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
to /SQL/json.
    18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5f745970{/SQL/json,null,AVAILABLE,@Spark}
    18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
to /SQL/execution.
    18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2afd8972{/SQL/execution,null,AVAILABLE,@Spark}
    18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
to /SQL/execution/json.
    18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5784f6b9{/SQL/execution/json,null,AVAILABLE,@Spark}
    18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
to /static/sql.
    18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6ccf06f1{/static/sql,null,AVAILABLE,@Spark}
    18/07/18 07:58:51 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    18/07/18 07:58:52 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator
endpoint
    18/07/18 07:58:52 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 254.142542 ms
    18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 65.397101 ms
    18/07/18 07:58:53 INFO spark.SparkContext: Starting job: show at <console>:24
    18/07/18 07:58:53 INFO scheduler.DAGScheduler: Got job 0 (show at <console>:24)
with 1 output partitions
    18/07/18 07:58:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (show at <console>:24)
    18/07/18 07:58:53 INFO scheduler.DAGScheduler: Parents of final stage: List()
    18/07/18 07:58:53 INFO scheduler.DAGScheduler: Missing parents: List()
    18/07/18 07:58:53 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4]
at show at <console>:24), which has no missing parents
    18/07/18 07:58:53 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory
(estimated size 8.1 KB, free 408.9 MB)
    18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 4.0 KB, free 408.9 MB)
    18/07/18 07:58:53 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on
ctr-e138-1518143905142-411342-01-000002.hwx.site:41073 (size: 4.0 KB, free: 408.9 MB)
    18/07/18 07:58:53 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1039
    18/07/18 07:58:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage
0 (MapPartitionsRDD[4] at show at <console>:24) (first 15 tasks are for partitions Vector(0))
    18/07/18 07:58:54 INFO cluster.YarnScheduler: Adding task set 0.0 with 1 tasks
    18/07/18 07:58:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0,
ctr-e138-1518143905142-411342-01-000008.hwx.site, executor 2, partition 0, PROCESS_LOCAL,
7864 bytes)
    [Stage 0:>                                                          (0 + 1) / 1]18/07/18
07:58:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ctr-e138-1518143905142-411342-01-000008.hwx.site:37671
(size: 4.0 KB, free: 366.3 MB)
    18/07/18 07:58:54 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    18/07/18 07:58:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0)
in 1115 ms on ctr-e138-1518143905142-411342-01-000008.hwx.site (executor 2) (1/1)
    18/07/18 07:58:55 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all
completed, from pool
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: ResultStage 0 (show at <console>:24)
finished in 1.609 s
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: Job 0 finished: show at <console>:24,
took 1.673348 s
    18/07/18 07:58:55 INFO spark.SparkContext: Starting job: show at <console>:24
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: Got job 1 (show at <console>:24)
with 1 output partitions
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (show at <console>:24)
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: Parents of final stage: List()
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: Missing parents: List()
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[4]
at show at <console>:24), which has no missing parents
    18/07/18 07:58:55 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory
(estimated size 8.1 KB, free 408.9 MB)
    18/07/18 07:58:55 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in
memory (estimated size 4.0 KB, free 408.9 MB)
    18/07/18 07:58:55 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on
ctr-e138-1518143905142-411342-01-000002.hwx.site:41073 (size: 4.0 KB, free: 408.9 MB)
    18/07/18 07:58:55 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
    18/07/18 07:58:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage
1 (MapPartitionsRDD[4] at show at <console>:24) (first 15 tasks are for partitions Vector(1))
    18/07/18 07:58:55 INFO cluster.YarnScheduler: Adding task set 1.0 with 1 tasks
    18/07/18 07:58:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1,
ctr-e138-1518143905142-411342-01-000007.hwx.site, executor 1, partition 1, PROCESS_LOCAL,
7864 bytes)
    18/07/18 07:58:55 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on
ctr-e138-1518143905142-411342-01-000007.hwx.site:33073 (size: 4.0 KB, free: 366.3 MB)
    18/07/18 07:58:55 INFO yarn.Client: Application report for application_1531383843352_0013
(state: RUNNING)
    [Stage 1:>                                                          (0 + 1) / 1]18/07/18
07:58:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 948 ms on
ctr-e138-1518143905142-411342-01-000007.hwx.site (executor 1) (1/1)
    18/07/18 07:58:56 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose tasks have all
completed, from pool
    18/07/18 07:58:56 INFO scheduler.DAGScheduler: ResultStage 1 (show at <console>:24)
finished in 0.956 s
    18/07/18 07:58:56 INFO scheduler.DAGScheduler: Job 1 finished: show at <console>:24,
took 0.959303 s
    +---+
    | id|
    +---+
    |  0|
    |  1|
    |  2|
    |  3|
    |  4|
    |  5|
    |  6|
    |  7|
    |  8|
    |  9|
    +---+
    ```
    
    Considering it's info level, I was thinking it's not super noisy. I am okay with disabling
though. I got your point. Let me defer this to @vanzin and @jerryshao.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message