hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chang Chen <baibaic...@gmail.com>
Subject Deal with viewFS in bulkLoadStoreFile
Date Wed, 09 Dec 2015 09:43:19 GMT
Hi Guys

As far as i know, it is better to use hdfs filesystem in HBase even if
federation is enabled.  This caused a minor performance issue in
the bulkLoadStoreFile, see the following code

  Path bulkLoadStoreFile(final String familyName, Path srcPath, long seqNum)
      throws IOException {
    FileSystem srcFs = ...  // viewFS
    FileSystem desFs = ... // HDFS


    // TODO deal with viewFS
    if (!FSHDFSUtils.isSameHdfs(conf, srcFs, desFs)) {
      // Copy the file if it's on another filesystem
    }
    ..
  }

In our case, srcFs is viewFs and desFs is hdfs, and hence
FSHDFSUtils.isSameHdfs will return false. However, both srcFs and desFs
refer to the same NameNode, which means if we can resolve them before or in
the *isSameHdfs*, we can avoid this extra copy, and improve performance.

any ideas?

Thanks
Chang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message