hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
Date Thu, 26 Sep 2013 05:28:03 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778489#comment-13778489
] 

Robert Kanter commented on MAPREDUCE-5530:
------------------------------------------

I was talking to [~sandyr] offline and he had an idea: {{mapred.lib.CombineFileInputFormat}}
would override {{isSplitable(JobContext context, Path file)}} to call {{isSplitable(FileSystem
fs, Path file)}}.  I believe this handles all situations except the one where the subclass
overrides both versions of {{isSplitable}}, but that's not likely to be common, especially
because {{isSplitable(JobContext context, Path file)}} isn't in branch-1 anyway.  

I think this is the best compromise and will solve most of the possible issues here.  Though,
it would look pretty ugly:
{code:java}
@Override
protected boolean isSplitable(JobContext context, Path file) {
  try {
    return isSplitable(FileSystem.get(context.getConfiguration()), file);
  }
  catch (IOException ioe) {
      throw new RuntimeException(ioe);
  }
}
{code}

What do you think?
                
> Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1
and branch-2
> ----------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5530
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv1, mrv2
>    Affects Versions: 2.1.1-beta
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>            Priority: Blocker
>         Attachments: MAPREDUCE-5530.patch
>
>
> {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
> {code:java}
> protected boolean isSplitable(FileSystem fs, Path file)
> {code}
> In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of {{mapreduce.lib.input.CombineFileInputFormat}},
from which it inherits the similar method:
> {code:java}
> protected boolean isSplitable(JobContext context, Path file)
> {code}
> This means that any code that subclasses {{mapred.lib.CombineFileInputFormat}} and does
not provide its own implementation of {{protected boolean isSplitable(FileSystem fs, Path
file)}} will not be binary or source compatible if it tries to call {{isSplitable}} with a
{{FileSystem}} argument anywhere (that is, if compiled against branch-1, it will throw a {{NoSuchMethodError}}
if run against branch-2; also, it won't even compile against branch-2).  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message