hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Rutman <nrut...@gmail.com>
Subject Re: Latency and speed of HDFS
Date Sat, 29 Jan 2011 17:58:45 GMT

On Jan 28, 2011, at 9:53 PM, Harsh J wrote:

> Moving discussion to hdfs-user mailing list: hdfs-user@hadoop.apache.com
> 
> On Sat, Jan 29, 2011 at 5:06 AM, Pei HE <peihe0@gmail.com> wrote:
>> Hi all,
>> I want to know the detailed performance of Hadoop.
>> 
>> I am writing a client code to test the latency, transfer speed of
>> HDFS, and the time of initiating a JVM for a task.
>> 
>> Does anyone know that which classes can be called without running a
>> whole map/reduce job to find out the performance of Hadoop?

You can get the performance of HDFS without Hadoop by using it directly:

######## Creating 1GB file ###########

#ftg-007:/usr/local/hadoop # dd if=/dev/zero of=/hdfs/demofile bs=1M count=1000
1000+0 records in                                                             
1000+0 records out                                                            
1048576000 bytes (1.0 GB) copied, 8.78527 s, 119 MB/s

######### Copy from local to HDFS ########
#ftg-007:/usr/local/hadoop # time ./bin/hadoop dfs  -copyFromLocal /hdfs/demofile /
real    0m51.911s
user    0m9.945s
sys     0m2.128s

######### Copy to Local from HDFS ########
#ftg-007:/usr/local/hadoop # time ./bin/hadoop dfs  -copyToLocal  /demofile /hdfs/test/
real    0m19.068s
user    0m8.205s 
sys     0m4.948s 

>> 
>> I am trying to create a DFSClient in the following way.
>>                Configuration conf = new Configuration();
>>                DFSClient client = new DFSClient(conf);
>>                client.exists("hdfs://localhost:9900/home/");
>> But, a RemoteException was thrown.
> 
> Would help to know what the cause for the throw of a RemoteException was.
> 
> -- 
> Harsh J
> www.harshj.com


Mime
View raw message