hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@apache.org>
Subject Re: Hadoop cluster network requirement
Date Mon, 01 Aug 2011 02:16:29 GMT

On Jul 31, 2011, at 12:08 PM, <jonathan.hwang@accenture.com>
 <jonathan.hwang@accenture.com> wrote:

> I was asked by our IT folks if we can put hadoop name nodes storage using a shared disk
storage unit.  

	What do you mean by "shared disk storage unit"?  There are lots of products out there that
would claim this, so actual deployment semantics are important.

> Does anyone have experience of how much IO throughput is required on the name nodes?

	IO throughput is completely dependent dependent upon how many changes are being applied to
the file system and frequency of edits log merging.  In the majority of cases it is "not much".
 What tends to happen where the storage is shared (such as a NAS) is that the *other* traffic
blocks the writes for too long because it is overloaded and the NN declares it dead.

>  What are the latency/data throughput requirements between the master and data nodes
- can this tolerate network routing?

	If you mean "different data centers", then no.  If you mean "same data center, but with routers
in between", then probably yes, but you add several more failure points, so this isn't recommended.

> Did anyone published any throughput requirement for the best network setup recommendation?
	Not that I know of.  It is very much dependent upon the actual workload being performed.
 But I wouldn't deploy anything slower than a 1:4 overcommit (uplink-to-host) on the DN side
for anything real/significant.

> This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the sender
immediately and delete the original. Any other use of the email by you is prohibited.

	Lawyers are funny people.  I wonder how much they got paid for this one.
View raw message