hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brahma Reddy Battula <brahmareddy.batt...@huawei.com>
Subject RE: NNBench on external HDFS
Date Thu, 23 Jul 2015 07:55:59 GMT
HI Alex

HDFS-7651 is raised for same..Please have a look at once..

Thanks & Regards
 Brahma Reddy Battula
_______________________________________
From: Alexander Striffeler [a.striffeler@students.unibe.ch]
Sent: Thursday, July 23, 2015 1:00 PM
To: user@hadoop.apache.org
Subject: Re: NNBench on external HDFS

Hi Chris,

Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...

Have a good day,
--Alex

On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool.  It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support.  Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test.  I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml.  Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath.  Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting.  If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it.  Feel free to post a patch too
> if you're inclined.  Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a.striffeler@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex


Mime
View raw message