crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-200) Move the logic for handling job outputs from CrunchJobHooks into the PathTarget objects
Date Tue, 30 Apr 2013 01:48:16 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645137#comment-13645137
] 

Micah Whitacre commented on CRUNCH-200:
---------------------------------------

Changes look good.  One little tweak might be..

{code}
  public void handleOutputs(Configuration conf, Path workingPath, int index,
      boolean mapOnlyJob) throws IOException {
    FileSystem srcFs = workingPath.getFileSystem(conf);
    Path srcPattern = getSourcePattern();
    Path[] srcs = FileUtil.stat2Paths(srcFs.globStatus(src), src);
    FileSystem dstFs = FileSystem.get(conf);
    if (!dstFs.exists(path)) {
      dstFs.mkdirs(path);
    }
    boolean sameFs = isCompatible(srcFs, path);
    for (Path s : srcs) {
      Path d = getDestFile(conf, s, path, mapOnlyJob);
      if (sameFs) {
        srcFs.rename(s, d);
      } else {
        FileUtil.copy(srcFs, s, dstFs, d, true, true, conf);
      }
    }
  }

protected Path getSourcePattern(){
  return new Path(workingPath, PlanningParameters.MULTI_OUTPUT_PREFIX + index + "-*");
}
{code}


                
> Move the logic for handling job outputs from CrunchJobHooks into the PathTarget objects
> ---------------------------------------------------------------------------------------
>
>                 Key: CRUNCH-200
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-200
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Josh Wills
>         Attachments: CRUNCH-200.patch
>
>
> Certain output formats, like Trevni, have a non-standard output file format that requires
custom handling logic. Currently, the logic for moving the output of a Crunch job from a temporary
place on the filesystem to its final output directory is handled by logic in CrunchJobHooks,
but the actual work of moving the files should be delegated to the Target object, which will
be aware of how the files are laid out in the filesystem and can move them appropriately.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message