hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Esteban Gutierrez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15860) Improvements for HBASE-14280 - Fixing Bulkload for HDFS HA Clusters
Date Thu, 09 Jun 2016 22:32:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323518#comment-15323518
] 

Esteban Gutierrez commented on HBASE-15860:
-------------------------------------------

Thanks for the patch [~wchevreuil]! since DFSUtil.getRpcAddressesForNameserviceId has been
around for a while it should be ok just to call directly instead of using a class loader.

> Improvements for HBASE-14280 - Fixing Bulkload for HDFS HA Clusters
> -------------------------------------------------------------------
>
>                 Key: HBASE-15860
>                 URL: https://issues.apache.org/jira/browse/HBASE-15860
>             Project: HBase
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 1.0.0
>            Reporter: Wellington Chevreuil
>            Priority: Minor
>         Attachments: HBASE-15860.patch
>
>
> HBASE-14280 introduced fix for bulkload failures when referring a remote cluster name
service id if "bulkloading" from a HA cluster.
> HBASE-14280 solution on *FSHDFSUtils.getNNAddresses* was to invoke *DFSUtil.getNNServiceRpcAddressesForCluster*
instead of *DFSUtil.getNNServiceRpcAddresses*. This works for hadoop 2.6 and above.
> Proposed change here is to use "*DFSUtil.getRpcAddressesForNameserviceId*" instead, which
already returns only addresses for specific nameservice informed. This is available since
hadoop 2.4.
> Sample proposal on FSHDFSUtils.getNNAddresses:
> ...
> {noformat}
>      String nameServiceId = serviceName.split(":")[1];
>         if (dfsUtilClazz == null) {
>           dfsUtilClazz = Class.forName("org.apache.hadoop.hdfs.DFSUtil");
>         }
>         if (getNNAddressesMethod == null) {
>           getNNAddressesMethod =
>               dfsUtilClazz.getMethod("getRpcAddressesForNameserviceId", Configuration.class,
>                 String.class, String.class);
>         }
>         Map<String, InetSocketAddress> nnMap =
>             (Map<String, InetSocketAddress>) getNNAddressesMethod
>                         .invoke(null, conf, nameServiceId, null);
>         for (Map.Entry<String, InetSocketAddress> e2 : nnMap.entrySet()) {
>             InetSocketAddress addr = e2.getValue();
>             addresses.add(addr);
>         }
> ...
> {noformat}
> Will also add test conditions for *FSHDFSUtils.isSameHdfs* to verify scenario when multiple
name service ids are defined.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message