hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: NNBench on external HDFS
Date Wed, 22 Jul 2015 18:14:56 GMT
Hi Alexander,

Your NNBench usage looks basically correct, but NNBench is not a standard
Hadoop tool.  It does not implement the org.apache.hadoop.util.Tool
interface, it does not execute through org.apache.hadoop.util.ToolRunner,
and therefore it does not support the command line arguments that a lot of
other Hadoop tools like the FsShell support.  Specifically, it does not
support passing -D arguments to override fs.defaultFS or any other
configuration properties.

An alternative way to handle this would be to get a local copy of the
configuration directory from the remote cluster that you want to test.  I
expect those configuration files would have fs.defaultFS set to the URL of
that remote cluster in core-site.xml.  Before launching NNBench, run
"export HADOOP_CONF_DIR=<path to local copy of configuration files>".
After exporting that environment variable, you can run "hadoop classpath"
to print the classpath that will be used by all "hadoop" commands and
confirm that the correct configuration directory for the target cluster is
on the classpath.  Then, you can run NNBench again, but drop the -D
argument, since it's going to get ignored anyway.

I don't see any reason why NNBench shouldn't implement the standard Tool
interface and thus support the command line arguments that you were
expecting.  If you'd like to request that as an enhancement, please go
ahead and file an HDFS jira to request it.  Feel free to post a patch too
if you're inclined.  Otherwise, someone else in the community can pick it
up.

I hope this helps.

--Chris Nauroth




On 7/22/15, 12:41 AM, "Alexander Striffeler"
<a.striffeler@students.unibe.ch> wrote:

>Hi all
>
>I'm pretty new to the Hadoop environment and I'm about performing some
>micro benchmarks. In particular, I'm struggling with executing NNBench
>against an external File System:
>
>hadoop jar 
>/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>ests.jar 
>nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>-baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>
>yields in
>java.lang.IllegalArgumentException: Wrong FS:
>hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>expected: hdfs://<native fs>
>
>If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>the -D option and writes the files to the native DFS. Does anyone have
>an idea how to solve this and nnbench an external DFS?
>
>Thanks a lot, any hints are very appreciated!
>Regards,
>Alex


Mime
View raw message