mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mallik Singaraju (JIRA)" <>
Subject [jira] [Commented] (MESOS-4087) Introduce a module for logging executor/task output
Date Fri, 17 Jun 2016 13:50:05 GMT


Mallik Singaraju commented on MESOS-4087:

Hi, we are using spark 1.6.1 deployed and running on mesos and I need some info on how to
capture the log of spark  executors running on mesos in spark 1.6.1. We are not using the
container based approach to deploy spark on mesos. Instead we are currently just deploying
spark job (<myapp>.jar) though spark-submit. 

I am not currently able to override the default behavior of spark executors always picking
up the  in the <SPARK_HOME>/conf .
I tried setting the log4j.configuration to  in the classpath of <my-application>.jar
and did supply that as argument to spark-submit. That does not seem to capture any logs of
the spark executor tasks in mesos. I did figure out that you worked on the logging piece through
JIRA. Do you have any recommendation on how to approach this?

> Introduce a module for logging executor/task output
> ---------------------------------------------------
>                 Key: MESOS-4087
>                 URL:
>             Project: Mesos
>          Issue Type: Task
>          Components: containerization, modules
>            Reporter: Joseph Wu
>            Assignee: Joseph Wu
>              Labels: logging, mesosphere
>             Fix For: 0.27.0
> Existing executor/task logs are logged to files in their sandbox directory, with some
nuances based on which containerizer is used (see background section in linked document).
> A logger for executor/task logs has the following requirements:
> * The logger is given a command to run and must handle the stdout/stderr of the command.
> * The handling of stdout/stderr must be resilient across agent failover.  Logging should
not stop if the agent fails.
> * Logs should be readable, presumably via the web UI, or via some other module-specific

This message was sent by Atlassian JIRA

View raw message