spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19829) The log about driver should support rolling like executor
Date Tue, 07 Mar 2017 10:55:37 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15899228#comment-15899228
] 

Sean Owen commented on SPARK-19829:
-----------------------------------

Why wouldn't log4j be a good solution here? its purpose is managing logs and can define rolling
appenders.

> The log about driver should support rolling like executor
> ---------------------------------------------------------
>
>                 Key: SPARK-19829
>                 URL: https://issues.apache.org/jira/browse/SPARK-19829
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: hustfxj
>            Priority: Minor
>
> We should rollback the log of the driver , or the log maybe large!!! 
> {code:title=DriverRunner.java|borderStyle=solid}
> // modify the runDriver
>   private def runDriver(builder: ProcessBuilder, baseDir: File, supervise: Boolean):
Int = {
>     builder.directory(baseDir)
>     def initialize(process: Process): Unit = {
>       // Redirect stdout and stderr to files------ the old code
> //      val stdout = new File(baseDir, "stdout")
> //      CommandUtils.redirectStream(process.getInputStream, stdout)
> //
> //      val stderr = new File(baseDir, "stderr")
> //      val formattedCommand = builder.command.asScala.mkString("\"", "\" \"", "\"")
> //      val header = "Launch Command: %s\n%s\n\n".format(formattedCommand, "=" * 40)
> //      Files.append(header, stderr, StandardCharsets.UTF_8)
> //      CommandUtils.redirectStream(process.getErrorStream, stderr)
>       // Redirect its stdout and stderr to files---------support rolling
>       val stdout = new File(baseDir, "stdout")
>       stdoutAppender = FileAppender(process.getInputStream, stdout, conf)
>       val stderr = new File(baseDir, "stderr")
>       val formattedCommand = builder.command.asScala.mkString("\"", "\" \"", "\"")
>       val header = "Launch Command: %s\n%s\n\n".format(formattedCommand, "=" * 40)
>       Files.append(header, stderr, StandardCharsets.UTF_8)
>       stderrAppender = FileAppender(process.getErrorStream, stderr, conf)
>     }
>     runCommandWithRetry(ProcessBuilderLike(builder), initialize, supervise)
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message