hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tonci Buljan <tonci.bul...@gmail.com>
Subject Please help...
Date Tue, 13 Jul 2010 10:04:48 GMT
Please help me, I can't figure out how to fix this problem.
I have a cluster of virtual machines under VMWare (windows XP is original
OS):

Ubuntu 8.10
Intel Pentium DUAL CPU E2180 @ 2 GHZ
Memory 1024 MB

I have a namenode and 8 more datanodes.
I want to start teragen and terasort programs and do a benchmark analysis of
a cluster running  1, 3 and all 8 datanodes.
Datanodes have only 20GB configured HDFS capacity each, so it is a total of
cca 150GB total.
I have no problem generating the input data with 2 or 8 maps but problem
comes out with terasort. When it comes to reduce phase, it generates a
following error:

10/07/13 10:59:40 INFO mapred.JobClient: Task Id :
attempt_201007131052_0002_r_000000_0, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.


As I understand I have to setup these parameters in mapred-site.xml to
override default values:

<property>
  <name>mapred.map.tasks</name>
  <value>?</value>
</property>
<property>
  <name>mapred.reduce.tasks</name>
  <value>?</value>
</property>

Does anyone know how to setup number of reducers so that it works :).

Thank you...

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message