hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CubicDesign <cubicdes...@gmail.com>
Subject Re: Processing 10MB files in Hadoop
Date Sat, 28 Nov 2009 00:07:55 GMT
Ok. I have set the number on maps to about 1760 (11 nodes * 16 
cores/node * 10 as recommended by Hadoop documentation) and my job still 
takes several hours to run instead of one.

Can be the overhead added by Hadoop that big? I mean I have over 30000 
small tasks (about one minute), each one starting its own JVM.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message