hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajith S (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (MAPREDUCE-6450) mapreduce.job.log4j-properties-file does not work for files on classpath
Date Thu, 13 Aug 2015 14:36:48 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ajith S reassigned MAPREDUCE-6450:

    Assignee: Ajith S

> mapreduce.job.log4j-properties-file does not work for files on classpath
> ------------------------------------------------------------------------
>                 Key: MAPREDUCE-6450
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6450
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Ben Podgursky
>            Assignee: Ajith S
>            Priority: Minor
> JobSubmitter.addLog4jToDistributedCache checks whether MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE
is set, and tries to put it in the distributed cache if it is set:
> {code}
>     String log4jPropertyFile =
>         conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, "");
>     if (!log4jPropertyFile.isEmpty()) {
>       short replication = (short)conf.getInt(Job.SUBMIT_REPLICATION, 10);
>       copyLog4jPropertyFile(job, jobSubmitDir, replication);
> {code}
> This throws an exception inside JobSubmitter.validateFilePath if the file does not exist
on the local filesystem, but is instead on the classpath.  This is unnecessarily restrictive,
because later on in MRApps.addLog4jSystemProperties, the actual configuration of -Dlog4j.configuration
works fine if the file is on the classpath and not a physical file:
> {code}
>     String log4jPropertyFile =
>         conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, "");
>     if (log4jPropertyFile.isEmpty()) {
>       vargs.add("-Dlog4j.configuration=container-log4j.properties");
> {code}
> (the default file container-log4j.properties is on the classpath anyway)
> This is a nuisance because we have a bunch of projects and it's much easier to distribute
shared configuration files via jars and dependencies than raw files.  It's not a blocker because
I'm able to work around it by setting -Dlog4j.configuration in yarn.app.mapreduce.am.command-opts,
but it's certainly hackier. 

This message was sent by Atlassian JIRA

View raw message