Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Date: Thu, 19 May 2016 17:37:13 +0000 (UTC)
From: "Wellington Chevreuil (JIRA)" <jira@apache.org>
To: dev@hbase.apache.org
Message-ID: <JIRA.12971218.1463679404000.238633.1463679433021@Atlassian.JIRA>
In-Reply-To: <JIRA.12971218.1463679404000@Atlassian.JIRA>
References: <JIRA.12971218.1463679404000@Atlassian.JIRA> <JIRA.12971218.1463679404684@arcas>
Subject: [jira] [Created] (HBASE-15860) Improvements for HBASE-14280 -
 Fixing Bulkload for HDFS HA Clusters
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 19 May 2016 17:37:15 -0000

Wellington Chevreuil created HBASE-15860:
--------------------------------------------

             Summary: Improvements for HBASE-14280 - Fixing Bulkload for HDFS HA Clusters
                 Key: HBASE-15860
                 URL: https://issues.apache.org/jira/browse/HBASE-15860
             Project: HBase
          Issue Type: Improvement
          Components: util
    Affects Versions: 1.0.0
            Reporter: Wellington Chevreuil
            Priority: Minor


HBASE-14280 introduced fix for bulkload failures when referring a remote cluster name service id if "bulkloading" from a HA cluster.

HBASE-14280 solution on *FSHDFSUtils.getNNAddresses* was to invoke *DFSUtil.getNNServiceRpcAddressesForCluster* instead of *DFSUtil.getNNServiceRpcAddresses*. This works for hadoop 2.6 and above.

Proposed change here is to use "*DFSUtil.getRpcAddressesForNameserviceId*" instead, which already returns only addresses for specific nameservice informed. This is available since hadoop 2.4.

Sample proposal on FSHDFSUtils.getNNAddresses:
...
{noformat}
     String nameServiceId = serviceName.split(":")[1];
        if (dfsUtilClazz == null) {
          dfsUtilClazz = Class.forName("org.apache.hadoop.hdfs.DFSUtil");
        }
        if (getNNAddressesMethod == null) {
          getNNAddressesMethod =
              dfsUtilClazz.getMethod("getRpcAddressesForNameserviceId", Configuration.class,
                String.class, String.class);
        }

        Map<String, InetSocketAddress> nnMap =
            (Map<String, InetSocketAddress>) getNNAddressesMethod
                        .invoke(null, conf, nameServiceId, null);
        for (Map.Entry<String, InetSocketAddress> e2 : nnMap.entrySet()) {
            InetSocketAddress addr = e2.getValue();
            addresses.add(addr);
        }
...
{noformat}

Will also add test conditions for *FSHDFSUtils.isSameHdfs* to verify scenario when multiple name service ids are defined.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)