hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wellington Chevreuil (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-15860) Improvements for HBASE-14280 - Fixing Bulkload for HDFS HA Clusters
Date Thu, 19 May 2016 17:37:13 GMT
Wellington Chevreuil created HBASE-15860:

             Summary: Improvements for HBASE-14280 - Fixing Bulkload for HDFS HA Clusters
                 Key: HBASE-15860
                 URL: https://issues.apache.org/jira/browse/HBASE-15860
             Project: HBase
          Issue Type: Improvement
          Components: util
    Affects Versions: 1.0.0
            Reporter: Wellington Chevreuil
            Priority: Minor

HBASE-14280 introduced fix for bulkload failures when referring a remote cluster name service
id if "bulkloading" from a HA cluster.

HBASE-14280 solution on *FSHDFSUtils.getNNAddresses* was to invoke *DFSUtil.getNNServiceRpcAddressesForCluster*
instead of *DFSUtil.getNNServiceRpcAddresses*. This works for hadoop 2.6 and above.

Proposed change here is to use "*DFSUtil.getRpcAddressesForNameserviceId*" instead, which
already returns only addresses for specific nameservice informed. This is available since
hadoop 2.4.

Sample proposal on FSHDFSUtils.getNNAddresses:
     String nameServiceId = serviceName.split(":")[1];
        if (dfsUtilClazz == null) {
          dfsUtilClazz = Class.forName("org.apache.hadoop.hdfs.DFSUtil");
        if (getNNAddressesMethod == null) {
          getNNAddressesMethod =
              dfsUtilClazz.getMethod("getRpcAddressesForNameserviceId", Configuration.class,
                String.class, String.class);

        Map<String, InetSocketAddress> nnMap =
            (Map<String, InetSocketAddress>) getNNAddressesMethod
                        .invoke(null, conf, nameServiceId, null);
        for (Map.Entry<String, InetSocketAddress> e2 : nnMap.entrySet()) {
            InetSocketAddress addr = e2.getValue();

Will also add test conditions for *FSHDFSUtils.isSameHdfs* to verify scenario when multiple
name service ids are defined.

This message was sent by Atlassian JIRA

View raw message