hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Libo Yu <yu_l...@hotmail.com>
Subject Is it a bug from CombineFileInputFormat?
Date Tue, 13 May 2014 02:10:44 GMT
Hi,

This is a private static inner class from CombineFilenputFormat.java
When "locations' length is 0", line M creates a new location which is 
immediately used at line N if the file is not splitable. Is that right?

  private static class OneFileInfo {
    private long fileSize;               // size of the file
    private OneBlockInfo[] blocks;       // all blocks in this file

    OneFileInfo(Path path, Configuration conf,
                boolean isSplitable,
                HashMap<String, List<OneBlockInfo>> rackToBlocks,
                HashMap<OneBlockInfo, String[]> blockToNodes,
                HashMap<String, List<OneBlockInfo>> nodeToBlocks,
                HashMap<String, Set<String>> rackToNodes,
                long maxSize)
                throws IOException {
      this.fileSize = 0;

      // get block locations from file system
      FileSystem fs = path.getFileSystem(conf);
      FileStatus stat = fs.getFileStatus(path);
      BlockLocation[] locations = fs.getFileBlockLocations(stat, 0, 
                                                           stat.getLen());
      // create a list of all block and their locations
      if (locations == null) {
        blocks = new OneBlockInfo[0];                                                
      } else {

        if(locations.length == 0) {
          locations = new BlockLocation[] { new BlockLocation() };  <-----------------------------M
        }

        if (!isSplitable) {
          // if the file is not splitable, just create the one block with
          // full file length
          blocks = new OneBlockInfo[1];
          fileSize = stat.getLen();
          blocks[0] = new OneBlockInfo(path, 0, fileSize, locations[0]     
              .getHosts(), locations[0].getTopologyPaths());            <----------------------------------N
        } else {
 		 	   		  
Mime
View raw message