Let’s suppose you are doing a read-intensive job like, for example, counting records. This is will be disk bandwidth limited. On a 4-node cluster with 2 local SATA on each node you should easily read 400MB/sec in aggregate. When you are running the Hadoop cluster, is the Hadoop processing co-located with the Ilsilon nodes? Is Hadoop configured to use OneFS or HDFS?
I’d like to pick the community brain on average throughput speeds for a moderately specced 4-node Hadoop cluster with 1GigE networking. Is it reasonable to expect constant average speeds of 150-200mb/sec on such setup? Forgive me if the question is loaded but we’re Hadoop cluster with HDFS served via EMC Isilon storage. We’re getting about 30mb/sec with our machines and we do not see a difference in job speed between 2 node cluster and 4 node cluster.