hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ayon Sinha <ayonsi...@yahoo.com>
Subject Re: Latency and speed of HDFS
Date Sat, 29 Jan 2011 06:22:03 GMT
You can run that code w/o running hadoop. As long as you have the libraries in 
classpath and the code is doing FileSystem operations, it will execute in 
standalone mode just fine. The last I remember the conf values from the client 
would be default values of the HDFS version and will not get picked up from the 
namenode installation. This might have changed, or may not. So it is in your 
best interest to verify what values is your client Conf object using. e.g. your 
innstallation might say rep factor 2, but default rep factor on Configuration 
object is 3, which can give you wrong perf numbers.

From: Harsh J <qwertymaniac@gmail.com>
To: hdfs-user@hadoop.apache.org; peihe0@gmail.com; zhengda1936@gmail.com
Sent: Fri, January 28, 2011 9:53:17 PM
Subject: Re: Latency and speed of HDFS

Moving discussion to hdfs-user mailing list: hdfs-user@hadoop.apache.com

On Sat, Jan 29, 2011 at 5:06 AM, Pei HE <peihe0@gmail.com> wrote:
> Hi all,
> I want to know the detailed performance of Hadoop.
> I am writing a client code to test the latency, transfer speed of
> HDFS, and the time of initiating a JVM for a task.
> Does anyone know that which classes can be called without running a
> whole map/reduce job to find out the performance of Hadoop?
> I am trying to create a DFSClient in the following way.
>                Configuration conf = new Configuration();
>                DFSClient client = new DFSClient(conf);
>                client.exists("hdfs://localhost:9900/home/");
> But, a RemoteException was thrown.

Would help to know what the cause for the throw of a RemoteException was.

Harsh J

View raw message