hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dito subandono" <dito.suband...@gmail.com>
Subject Smallest file size limit for hadoop to work faster than other application
Date Tue, 22 Jul 2008 17:31:42 GMT
i had a test with a log file analysis that was written with java and ran on
Hadoop.
i ran my log file analysis on a Intel Quad Core processor with 2 GB of
memory.
i set the map task to 40 and reduce task to 8.

the size of the log files i had test are 1GB to 4GB because i ran out of
storage resource.
i compare it with webalizer on a computer w/ Celeron processor and 256MB of
memory.

webalizer ran about 10x more faster.

what i did was just a small experiment and maybe still have to config things
more.
can anyone share the experience about the smallest file size limit for
hadoop to run faster than other application?

thanks..

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message