hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Artem Ervits" <are9...@nyp.org>
Subject RE: Hadoop throughput question
Date Thu, 03 Jan 2013 22:46:41 GMT
I have a 4.5gb file with records in SequenceFile format. If I use SequenceFile.Reader class
to count the records in this file, which amount to 5.5million records, it takes 176seconds,
or roughly 26mb/sec.

Thank you.

From: Michael Katzenellenbogen [mailto:michael@cloudera.com]
Sent: Thursday, January 03, 2013 5:08 PM
To: user@hadoop.apache.org
Subject: Re: Hadoop throughput question

Loaded question indeed.

How are you measuring that 30mb/s? Is that per machine / NIC? HDFS throughout? Some other
metric?

-Michael

On Jan 3, 2013, at 5:01 PM, Artem Ervits <are9004@nyp.org<mailto:are9004@nyp.org>>
wrote:
Hello all,

I'd like to pick the community brain on average throughput speeds for a moderately specced
4-node Hadoop cluster with 1GigE networking. Is it reasonable to expect constant average speeds
of 150-200mb/sec on such setup? Forgive me if the question is loaded but we're Hadoop cluster
with HDFS served via EMC Isilon storage. We're getting about 30mb/sec with our machines and
we do not see a difference in job speed between 2 node cluster and 4 node cluster.

Thank you.



--------------------



This electronic message is intended to be for the use only of the named recipient, and may
contain information that is confidential or privileged.  If you are not the intended recipient,
you are hereby notified that any disclosure, copying, distribution or use of the contents
of this message is strictly prohibited.  If you have received this message in error or are
not the named recipient, please notify us immediately by contacting the sender at the electronic
mail address noted above, and delete and destroy all copies of this message.  Thank you.



--------------------



This electronic message is intended to be for the use only of the named recipient, and may
contain information that is confidential or privileged.  If you are not the intended recipient,
you are hereby notified that any disclosure, copying, distribution or use of the contents
of this message is strictly prohibited.  If you have received this message in error or are
not the named recipient, please notify us immediately by contacting the sender at the electronic
mail address noted above, and delete and destroy all copies of this message.  Thank you.






--------------------

This electronic message is intended to be for the use only of the named recipient, and may
contain information that is confidential or privileged.  If you are not the intended recipient,
you are hereby notified that any disclosure, copying, distribution or use of the contents
of this message is strictly prohibited.  If you have received this message in error or are
not the named recipient, please notify us immediately by contacting the sender at the electronic
mail address noted above, and delete and destroy all copies of this message.  Thank you.




--------------------

This electronic message is intended to be for the use only of the named recipient, and may
contain information that is confidential or privileged.  If you are not the intended recipient,
you are hereby notified that any disclosure, copying, distribution or use of the contents
of this message is strictly prohibited.  If you have received this message in error or are
not the named recipient, please notify us immediately by contacting the sender at the electronic
mail address noted above, and delete and destroy all copies of this message.  Thank you.




Mime
View raw message