hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Unnecessary file copying during the bulkload: should we backport the fix in 0.96?
Date Fri, 07 Sep 2012 01:53:08 GMT
On Thu, Sep 6, 2012 at 5:30 PM, pig user <piguser@gmail.com> wrote:
> In HBase 0.94, bulkload would always copy the files:
>
> // Move the file if it's on another filesystem
> FileSystem srcFs = srcPath.getFileSystem(conf);
> if (!srcFs.equals(fs)) {
>    LOG.info("File " + srcPath + " on different filesystem than " +
>       "destination store - moving to this filesystem.");
>     ......
>
> Since fs here is an instance of HFileSystem. This would result in the
> load taking long time to complete if the HFiles are in the destination
> cluster.
>
> This is fixed in trunk:
>
> FileSystem srcFs = srcPath.getFileSystem(conf);
> FileSystem desFs = fs instanceof HFileSystem ?
> ((HFileSystem)fs).getBackingFs() : fs;
>     if (!srcFs.equals(desFs)) {
>     ... ...
>
> My question is: should we back port the fix to 0.94?

Sure, make an issue and backport the patch and we'll commit it.
St.Ack

Mime
View raw message