hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
Date Mon, 09 Mar 2015 23:39:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353892#comment-14353892
] 

Colin Patrick McCabe commented on HDFS-7847:
--------------------------------------------

DFSClient.java: this change adds three new fields to DFSClient.  But they only seem to be
used by unit tests.  It seems like we should just put these inside the unit test(s) that are
using these-- if necessary, by adding a helper method.  There's no reason to add more fields
to DFSClient.  Also remember that when using FileContext, we create new DFSClients all the
time.

{{this.datanodeProtocolProxy = NameNodeProxies.createProxy...}}

It seems kind of odd to have {{NameNodeProxies#createProxy}} create a proxy to the datanode.
 In general, when you see "NameNodeProxies" I think "proxies used by the NameNode" and this
doesn't fit with that.  Can you give a little more context about why this is a good idea (as
opposed to just having some custom code in the unit test or in a unit test util class that
creates a proxy)

{code}
121	    " [-remoteNamenode <namenode URI>]\n" +
122	    "     If using -remoteNamenode, set the namenode's" +
123	    "         dfs.namenode.fs-limits.min-block-size to 16.";
{code}
Of course the NameNode may or may not be remote here.  It seems like {{\-\-nnuri}} or just
{{\-\-namenode}} or something like that would be more descriptive.

{code}
378	      final int remoteNNIndex = args.indexOf("-remoteNamenode");
379	      if (remoteNNIndex >= 0) {
380	        if (args.size() <= remoteNNIndex + 1) {
381	          printUsage();
382	        }
383	        remoteNamenode = args.get(remoteNNIndex + 1);
384	        args.remove(remoteNNIndex + 1);
385	        args.remove(remoteNNIndex);
386	      }
{code}

Instead of this boilerplate, just use {{StringUtils#popOptionWithArgument}}

{code}
-          replication, BLOCK_SIZE, null);
+          replication, BLOCK_SIZE, CryptoProtocolVersion.supported());
{code}

This fix is a little bit separate, right?  I suppose we can do it in this JIRA, though.

> Modify NNThroughputBenchmark to be able to operate on a remote NameNode
> -----------------------------------------------------------------------
>
>                 Key: HDFS-7847
>                 URL: https://issues.apache.org/jira/browse/HDFS-7847
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7836
>            Reporter: Colin Patrick McCabe
>            Assignee: Charles Lamb
>         Attachments: HDFS-7847.000.patch, make_blocks.tar.gz
>
>
> Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A
followon Jira will modify it some more to allow quantifying native and java heap sizes, and
some latency numbers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message