spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-15891) Make YARN logs less noisy
Date Fri, 02 Sep 2016 22:33:20 GMT


Apache Spark commented on SPARK-15891:

User 'vanzin' has created a pull request for this issue:

> Make YARN logs less noisy
> -------------------------
>                 Key: SPARK-15891
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 2.0.0
>            Reporter: Marcelo Vanzin
>            Priority: Minor
> Spark can generate a lot of logs when running in YARN mode. The problem is already a
little bad with normal configuration, but is even worse with dynamic allocation on.
> The first problem is that for every executor Spark launches, it will print the whole
command and all the env variables it's setting, even though those are exactly the same for
every executor. That's not too bad with a handful of executors, but get annoying pretty soon
after that. Dynamic allocation makes that problem worse since there executors constantly being
started and then going away.
> Also, there's a lot of logging generated by the dynamic allocation backend code in the
YARN module. We should audit those and make sure they all make sense, and whether / how to
reduce the amount of logging.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message