hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
Date Wed, 14 Feb 2018 15:32:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364275#comment-16364275
] 

Jason Lowe commented on HADOOP-15227:
-------------------------------------

Yeah, mapred-default and mapred-site aren't loaded until the JobConf class is loaded.  A common
mistake for code is to create a plain Configuration early in {{main}} and try to lookup mapred
properties (or even hdfs or yarn properties) expecting to get the default if they are not
set by the user.  The easy fix is to create a JobConf instead of a Configuration if the code
knows it wants to do mapred stuff.


> add mapreduce.outputcommitter.factory.scheme.s3a to core-default
> ----------------------------------------------------------------
>
>                 Key: HADOOP-15227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15227
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Blocker
>
> Need to add this property to core-default.xml. It's documented as being there, but it
isn't.
> {code}
> <property>
>   <name>mapreduce.outputcommitter.factory.scheme.s3a</name>
>   <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value>
>   <description>
>     The committer factory to use when writing data to S3A filesystems.
>   </description>
> </property>
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message