hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6956) FileOutputCommitter to gain abstract superclass PathOutputCommitter
Date Fri, 15 Sep 2017 15:42:01 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168039#comment-16168039
] 

Hadoop QA commented on MAPREDUCE-6956:
--------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 14s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 41s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 22s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 16s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 26s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 43s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 18s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 21s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 21s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 21s{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 3
new + 162 unchanged - 0 fixed = 165 total (was 162) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 15s{color}
| {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 93 unchanged - 12 fixed = 93 total (was 105) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 24s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 51s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 16s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 33s{color} | {color:green}
hadoop-mapreduce-client-core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 11s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 47s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6956 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12887361/MAPREDUCE-6956-002.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux f1b77e584af6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 78bdf10 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7139/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7139/testReport/
|
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7139/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> FileOutputCommitter to gain abstract superclass PathOutputCommitter
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6956
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6956
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 3.0.0-beta1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: MAPREDUCE-6956-001.patch, MAPREDUCE-6956-002.patch
>
>
> This is the initial step of MAPREDUCE-6823, which proposes a factory behind {{FileOutputFormat}}
to create different committers for different filesystems, if so configured..
> This patch simply adds the new abstract superclass of {{FileOutputCommitter}}, {{PathOutputCommitter
extends OutputCommitter}}. This abstract class adds the {{getWorkPath()}} method as an abstract
method, with {{FIleOutputCommitter}} being the implementation..
> {{FileOutputFormat}} then relaxes its requirement of any committer returned by {{getOutputCommitter()}},
so that instead of requiring a  {{FileOutputCommitter}} or subclass, it only needs a {{PathOutputCommitter}},
using {{PathOutputCommitter.getWorkPath()}} to get the work path.
> What does that do?
> It allows people to implement subclasses of {{FileOutputFormat}} which can provide their
own committers *which don't need to inherit the complexity that FileOutputCommitter has acquired
over time*
> Currently anyone implementing a new committer (example: Netflix S3 committer) needs to
subclass {{FileOutputCommitter}}, which is too complex to understand except under a debugger
with co-recursive routines, lots of methods which need to be overwritten to guarantee a safe
subclass, and, because of its critical role and known subclassing, something which isn't ever
going to be cleaned up.
> A new, lean, parent class which {{FileOutputFormat}} can handle allows people to write
new committers which don't have to worry about implementation details of {{FileOutputCommitter}},
but instead how well they implement the semantics of committing work.
> The full MAPREDUCE-6823 goes beyond this with a change to {{FileOutputFormat}} for a
factory for creating FS-specific {{PathOutputCommitter}} instances. This patch doesn't include
that, as that is something which needs to be reviewed in the context of HADOOP-13786 and ideally
1+ committer for another store, so people can say "this factory model works".
> All I'm proposing here is: tune the committer class hierarchy in MRv2 so that people
can more easily implement committers, and when that factory is done, for it to be switched
to easily. And I'd like this in branch-3 from the outset, so existing code which calls {{FileOutputFormat.getCommitter()}}
to get a {{FileOutputCommitter}} *just to call getWorkPath()* can move to the new interface
across all of Hadoop 3.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message