hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dito subandono" <dito.suband...@gmail.com>
Subject Smallest file size limit for hadoop to work faster than other application
Date Tue, 22 Jul 2008 17:31:42 GMT
i had a test with a log file analysis that was written with java and ran on
i ran my log file analysis on a Intel Quad Core processor with 2 GB of
i set the map task to 40 and reduce task to 8.

the size of the log files i had test are 1GB to 4GB because i ran out of
storage resource.
i compare it with webalizer on a computer w/ Celeron processor and 256MB of

webalizer ran about 10x more faster.

what i did was just a small experiment and maybe still have to config things
can anyone share the experience about the smallest file size limit for
hadoop to run faster than other application?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message