Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Sun, 2 Nov 2014 04:42:34 +0000 (UTC)
From: "Junping Du (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12736697.1409022935000.393053.1414903354935@Atlassian.JIRA>
In-Reply-To: <JIRA.12736697.1409022935000@Atlassian.JIRA>
References: <JIRA.12736697.1409022935000@Atlassian.JIRA>
 <JIRA.12736697.1409022935620@arcas>
Subject: [jira] [Commented] (MAPREDUCE-6052) Support overriding
 log4j.properties per job
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193689#comment-14193689 ] 

Junping Du commented on MAPREDUCE-6052:
---------------------------------------

Thanks [~vinodkv] for comments!
bq. Users can use URI fragments in distributed-cache - for e.g. hdfs://a/b/c#symlink. Either we should explicitly reject fragments or support them. See MRApps.parseDistributedCacheArtifacts() for similar changes that we do for regular dist-cache files.
IMO, URI fragment doesn't make too much sense here as we are hiding details for how log4j.properties upload to hdfs and get download to each node's local cache and put it on classpath. MR over distributed cache support fragment as user need to explicitly set class path that included in tar ball and symlink for directory is easily to reference. In this case, user only need to set one value here to point to local filesystem. If you still think fragment is required, please file a separated JIRA instead of reopening this.

bq. If the user passes his own log4j file, setting YARN_APP_CONTAINER_LOG_SIZE, YARN_APP_CONTAINER_LOG_BACKUPS and logLevel may not matter at all. But let's set them as you do now, just leave a code comment that setting them may or may not really take effect.
Agree we should put this on documentation.

bq. job.setWorkingDirectory() getting called repeatedly?
As Zhijie's raised above, we decided to address the consolidation of some code flow here in MAPREDUCE-6148.

bq. For the null-scheme input, "File " + file + " does not exist." -> "File " + file + " does not exist on the local file-system."
Nice catch! Will address this minor issue in refactoring JIRA.

bq. Delete this log statement that you might have added for your testing: LOG.debug("default FileSystem: " + jtFs.getUri());
I think it is helpful and prefer to leave it here which doesn't affect system runtime performance. Right?

bq. What happens if the user passes a relative path for MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE? I think it works. If so, let's put this in the documentation too.
Relative path will base on current user's working directory. Let's document it.

Other documentation comments make sense to me. Given most of comments are documentation, let's resolve this and file a separated document JIRA to track.

> Support overriding log4j.properties per job
> -------------------------------------------
>
>                 Key: MAPREDUCE-6052
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.5.0
>            Reporter: Junping Du
>            Assignee: Junping Du
>             Fix For: 2.6.0
>
>         Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch
>
>
> For current MR application, the "log4j.configuration" is hard coded to container-log4j.properties within each node. We still need flexibility to override it per job like what we do in MRV1.
> {code}
>   public static void addLog4jSystemProperties(
>       String logLevel, long logSize, int numBackups, List<String> vargs) {
>     vargs.add("-Dlog4j.configuration=container-log4j.properties");
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)