hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pig user <pigu...@gmail.com>
Subject Unnecessary file copying during the bulkload: should we backport the fix in 0.96?
Date Fri, 07 Sep 2012 00:30:01 GMT
In HBase 0.94, bulkload would always copy the files:

// Move the file if it's on another filesystem
FileSystem srcFs = srcPath.getFileSystem(conf);
if (!srcFs.equals(fs)) {
   LOG.info("File " + srcPath + " on different filesystem than " +
      "destination store - moving to this filesystem.");
    ......

Since fs here is an instance of HFileSystem. This would result in the
load taking long time to complete if the HFiles are in the destination
cluster.

This is fixed in trunk:

FileSystem srcFs = srcPath.getFileSystem(conf);
FileSystem desFs = fs instanceof HFileSystem ?
((HFileSystem)fs).getBackingFs() : fs;
    if (!srcFs.equals(desFs)) {
    ... ...

My question is: should we back port the fix to 0.94?

Mime
View raw message