hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rita <rmorgan...@gmail.com>
Subject Re: Latency and speed of HDFS
Date Sun, 30 Jan 2011 00:42:41 GMT
Are there any tips to reduce the latency in general for hdfs?

I noticed that when trying to copy a 30gb file it takes a "while". I know
its subjective but would like to know if anyone has any tricks to reduce
latency.




On Sat, Jan 29, 2011 at 12:58 PM, Nathan Rutman <nrutman@gmail.com> wrote:

>
> On Jan 28, 2011, at 9:53 PM, Harsh J wrote:
>
> > Moving discussion to hdfs-user mailing list: hdfs-user@hadoop.apache.com
> >
> > On Sat, Jan 29, 2011 at 5:06 AM, Pei HE <peihe0@gmail.com> wrote:
> >> Hi all,
> >> I want to know the detailed performance of Hadoop.
> >>
> >> I am writing a client code to test the latency, transfer speed of
> >> HDFS, and the time of initiating a JVM for a task.
> >>
> >> Does anyone know that which classes can be called without running a
> >> whole map/reduce job to find out the performance of Hadoop?
>
> You can get the performance of HDFS without Hadoop by using it directly:
>
> ######## Creating 1GB file ###########
>
> #ftg-007:/usr/local/hadoop # dd if=/dev/zero of=/hdfs/demofile bs=1M
> count=1000
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 8.78527 s, 119 MB/s
>
> ######### Copy from local to HDFS ########
> #ftg-007:/usr/local/hadoop # time ./bin/hadoop dfs  -copyFromLocal
> /hdfs/demofile /
> real    0m51.911s
> user    0m9.945s
> sys     0m2.128s
>
> ######### Copy to Local from HDFS ########
> #ftg-007:/usr/local/hadoop # time ./bin/hadoop dfs  -copyToLocal  /demofile
> /hdfs/test/
> real    0m19.068s
> user    0m8.205s
> sys     0m4.948s
>
> >>
> >> I am trying to create a DFSClient in the following way.
> >>                Configuration conf = new Configuration();
> >>                DFSClient client = new DFSClient(conf);
> >>                client.exists("hdfs://localhost:9900/home/");
> >> But, a RemoteException was thrown.
> >
> > Would help to know what the cause for the throw of a RemoteException was.
> >
> > --
> > Harsh J
> > www.harshj.com
>
>


-- 
--- Get your facts first, then you can distort them as you please.--

Mime
View raw message