spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (SPARK-7169) Allow to specify metrics configuration more flexibly
Date Wed, 29 Apr 2015 20:30:08 GMT


Apache Spark reassigned SPARK-7169:

    Assignee:     (was: Apache Spark)

> Allow to specify metrics configuration more flexibly
> ----------------------------------------------------
>                 Key: SPARK-7169
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.2.2, 1.3.1
>            Reporter: Jacek Lewandowski
>            Priority: Minor
> Metrics are configured in {{}} file. Path to this file is specified
in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is read when {{MetricsSystem}}
is created which means, during {{SparkEnv}} initialisation. 
> h5.Problem
> When the user runs his application he has no way to provide the metrics configuration
for executors. Although one can specify the path to metrics configuration file (1) the path
is common for all the nodes and the client machine so there is implicit assumption that all
the machines has same file in the same location, and (2) actually the user needs to copy the
file manually to the worker nodes because the file is read before the user files are populated
to the executor local directories. All of this makes it very difficult to play with the metrics
> h5. Proposed solution
> I think that the easiest and the most consistent solution would be to move the configuration
from a separate file directly to {{SparkConf}}. We may prefix all the configuration settings
from the metrics configuration by, say {{spark.metrics.props}}. For the backward compatibility,
these properties would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark 1.2 and Spark
> Appreciate any feedback.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message