hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Chandel <amitchan...@gmail.com>
Subject Re: File Transfer Rates
Date Tue, 10 Feb 2009 23:03:33 GMT
With my setup. I have been able to get  10MBps write speed, 40MBps read
speed  while writing multiple files (ranging a few Bytes to 100MB) into
SequenceFiles, and reading them back. The cluster has 1Gbps backbone.

On Tue, Feb 10, 2009 at 5:53 PM, Mark Kerzner <markkerzner@gmail.com> wrote:

> Brian, I have a similar question: why does transfer from a local filesystem
> to SequenceFile takes so long (about 1 second per Meg)?
> Thank you,
> Mark
>
> On Tue, Feb 10, 2009 at 4:46 PM, Brian Bockelman <bbockelm@cse.unl.edu
> >wrote:
>
> >
> > On Feb 10, 2009, at 4:10 PM, Wasim Bari wrote:
> >
> >  Hi,
> >>   Could someone help me to find some real Figures (transfer rate) about
> >> Hadoop File transfer  from local filesystem to HDFS, S3 etc and among
> >> Storage Systems (HDFS to S3 etc)
> >>
> >> Thanks,
> >>
> >> Wasim
> >>
> >
> > What are you looking for?  Maximum possible transfer rate?  Maximum
> > possible transfer rate per client?  Generally, if you're using the Java
> > client, transfer rate to/from HDFS is limited by the hardware you have
> and
> > the network connection (if you have 1Gbps per client).
> >
> > I could give you a graph showing a peak of 9Gbps from our Hadoop instance
> > to the WAN, but that's not very interesting if you don't have a 10Gbps
> > pipe...
> >
> > Brian
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message