hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Charles Lamb (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
Date Mon, 16 Mar 2015 14:16:39 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Charles Lamb updated HDFS-7847:
-------------------------------
    Attachment: HDFS-7847.002.patch

@cmccabe, @stack, thanks for the review!

bq. DFSClient.java: this change adds three new fields to DFSClient. But they only seem to
be used by unit tests. It seems like we should just put these inside the unit test(s) that
are using these-- if necessary, by adding a helper method. There's no reason to add more fields
to DFSClient. Also remember that when using FileContext, we create new DFSClients all the
time.

Good point. I've left the existing {code}ClientProtocol namenode{code} field alone. The other
3 proxies are created on-demand by their getters. That means no change in DFSClient instance
size.

bq. It seems kind of odd to have NameNodeProxies#createProxy create a proxy to the datanode.

It's actually a proxy to the NN for the DatanodeProtocol. That's the same protocol that the
DN uses to speak with the NN when it's sending (among other things) block reports. But with
some ideas from @stack, I got rid of the changes to NameNodeProxies.

bq. Of course the NameNode may or may not be remote here. It seems like --nnuri or just --namenode
or something like that would be more descriptive.

Yeah, I agree. I changed it to -namenode.

bq. Instead of this boilerplate, just use StringUtils#popOptionWithArgument.

Changed. I was just trying to match the existing code, but the using StringUtils is for the
better. 

{code}
-          replication, BLOCK_SIZE, null);
+          replication, BLOCK_SIZE, CryptoProtocolVersion.supported());
{code}

bq. This fix is a little bit separate, right? I suppose we can do it in this JIRA, though.

Without this, the relevant PBHelper.convert code throws NPE on the supportVersions arg.


> Modify NNThroughputBenchmark to be able to operate on a remote NameNode
> -----------------------------------------------------------------------
>
>                 Key: HDFS-7847
>                 URL: https://issues.apache.org/jira/browse/HDFS-7847
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7836
>            Reporter: Colin Patrick McCabe
>            Assignee: Charles Lamb
>         Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, make_blocks.tar.gz
>
>
> Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A
followon Jira will modify it some more to allow quantifying native and java heap sizes, and
some latency numbers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message