hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job
Date Sun, 02 Nov 2014 04:42:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193689#comment-14193689

Junping Du commented on MAPREDUCE-6052:

Thanks [~vinodkv] for comments!
bq. Users can use URI fragments in distributed-cache - for e.g. hdfs://a/b/c#symlink. Either
we should explicitly reject fragments or support them. See MRApps.parseDistributedCacheArtifacts()
for similar changes that we do for regular dist-cache files.
IMO, URI fragment doesn't make too much sense here as we are hiding details for how log4j.properties
upload to hdfs and get download to each node's local cache and put it on classpath. MR over
distributed cache support fragment as user need to explicitly set class path that included
in tar ball and symlink for directory is easily to reference. In this case, user only need
to set one value here to point to local filesystem. If you still think fragment is required,
please file a separated JIRA instead of reopening this.

bq. If the user passes his own log4j file, setting YARN_APP_CONTAINER_LOG_SIZE, YARN_APP_CONTAINER_LOG_BACKUPS
and logLevel may not matter at all. But let's set them as you do now, just leave a code comment
that setting them may or may not really take effect.
Agree we should put this on documentation.

bq. job.setWorkingDirectory() getting called repeatedly?
As Zhijie's raised above, we decided to address the consolidation of some code flow here in

bq. For the null-scheme input, "File " + file + " does not exist." -> "File " + file +
" does not exist on the local file-system."
Nice catch! Will address this minor issue in refactoring JIRA.

bq. Delete this log statement that you might have added for your testing: LOG.debug("default
FileSystem: " + jtFs.getUri());
I think it is helpful and prefer to leave it here which doesn't affect system runtime performance.

bq. What happens if the user passes a relative path for MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE?
I think it works. If so, let's put this in the documentation too.
Relative path will base on current user's working directory. Let's document it.

Other documentation comments make sense to me. Given most of comments are documentation, let's
resolve this and file a separated document JIRA to track.

> Support overriding log4j.properties per job
> -------------------------------------------
>                 Key: MAPREDUCE-6052
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.5.0
>            Reporter: Junping Du
>            Assignee: Junping Du
>             Fix For: 2.6.0
>         Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, MAPREDUCE-6052-v4.patch,
> For current MR application, the "log4j.configuration" is hard coded to container-log4j.properties
within each node. We still need flexibility to override it per job like what we do in MRV1.
> {code}
>   public static void addLog4jSystemProperties(
>       String logLevel, long logSize, int numBackups, List<String> vargs) {
>     vargs.add("-Dlog4j.configuration=container-log4j.properties");
> {code}

This message was sent by Atlassian JIRA

View raw message