hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Striffeler <a.striffe...@students.unibe.ch>
Subject Re: NNBench on external HDFS
Date Thu, 23 Jul 2015 07:30:17 GMT
Hi Chris,

Wow, thanks a lot for your swift and extensive response! I'll try your 
suggestion with the local copy and in a second step I'll open a jira 

Have a good day,

On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool.  It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support.  Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test.  I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml.  Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath.  Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting.  If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it.  Feel free to post a patch too
> if you're inclined.  Otherwise, someone else in the community can pick it
> up.
> I hope this helps.
> --Chris Nauroth
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a.striffeler@students.unibe.ch> wrote:
>> Hi all
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>> hadoop jar
>> /usr/hdp/
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex

View raw message