hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Nguyen <andrew-lists-had...@ucsfcti.org>
Subject Re: Optimal setup for a test problem
Date Mon, 12 Apr 2010 20:45:55 GMT
I don't think you can :-).  Sorry, they are 100Mbps NIC's...  I get 95Mbit/sec from one node
to another with iperf.

Should I still be expecting such dismal performance with just 100Mbps?

On Apr 12, 2010, at 1:31 PM, Todd Lipcon wrote:

> On Mon, Apr 12, 2010 at 1:05 PM, Andrew Nguyen <
> andrew-lists-hadoop@ucsfcti.org> wrote:
> 
>> 5 identically spec'ed nodes, each has:
>> 
>> 2 GB RAM
>> Pentium 4 3.0G with HT
>> 250GB HDD on PATA
>> 10Mbps NIC
>> 
> 
> This is probably your issue - 10mbps nic? I didn't know you could even get
> those anymore!
> 
> Hadoop runs on commodity hardware, but you're not likely to get reasonable
> performance with hardware like that.
> 
> -Todd
> 
> 
>> On Apr 12, 2010, at 11:58 AM, alex kamil wrote:
>> 
>>> Andrew,
>>> 
>>> I would also suggest to run DFSIO benchmark to isolate io related issues
>>> 
>>> hadoop jar hadoop-0.20.2-test.jar TestDFSIO -write -nrFiles 10 -fileSize
>> 1000
>>> hadoop jar hadoop-0.20.2-test.jar TestDFSIO -read -nrFiles 10 -fileSize
>> 1000
>>> 
>>> there are additional tests specific for mapreduce -  run  "hadoop jar
>> hadoop-0.20.2-test.jar" for the complete list
>>> 
>>> 45 min for mapping 6GB on 5 nodes is way too high assuming your
>> gain/offset conversion is a simple algebraic manipulation
>>> 
>>> it takes less than 5 min  to run a simple mapper (using streaming) on a
>> 4 nodes cluster on something like 10GB, the mapper i used was an awk command
>> extracting <key:value> pair from a log (no reducer)
>>> 
>>> Thanks
>>> Alex
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Apr 12, 2010 at 1:53 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>> Hi Andrew,
>>> 
>>> Do you need the sorting behavior that having an identity reducer gives
>> you?
>>> If not, set the number of reduce tasks to 0 and you'll end up with a map
>>> only job, which should be significantly faster.
>>> 
>>> -Todd
>>> 
>>> On Mon, Apr 12, 2010 at 9:43 AM, Andrew Nguyen <
>>> andrew-lists-hadoop@ucsfcti.org> wrote:
>>> 
>>>> Hello,
>>>> 
>>>> I recently setup a 5 node cluster (1 master, 4 slaves) and am looking
>> to
>>>> use it to process high volumes of patient physiologic data.  As an
>> initial
>>>> exercise to gain a better understanding, I have attempted to run the
>>>> following problem (which isn't the type of problem that Hadoop was
>> really
>>>> designed for, as is my understanding).
>>>> 
>>>> I have a 6G data file, that contains key/value of <sample number,
>> sample
>>>> value>.  I'd like to convert the values based on a gain/offset to their
>>>> physical units.  I've setup a MapReduce job using streaming where the
>> mapper
>>>> does the conversion, and the reducer is just an identity reducer.
>> Based on
>>>> other threads on the mailing list, my initial results are consistent in
>> the
>>>> fact that it takes considerably more time to process this in Hadoop
>> then it
>>>> is on my Macbook pro (45 minutes vs. 13 minutes).  The input is a
>> single 6G
>>>> file and it looks like the file is being split into 101 map tasks.
>> This is
>>>> consistent with the 64M block sizes.
>>>> 
>>>> So my questions are:
>>>> 
>>>> * Would it help to increase the block size to 128M?  Or, decrease the
>> block
>>>> size?  What are some key factors to think about with this question?
>>>> * Are there any other optimizations that I could employ?  I have looked
>>>> into LzoCompression but I'd like to still work without compression
>> since the
>>>> single thread job that I'm comparing to doesn't use any sort of
>> compression.
>>>> I know I'm comparing apples to pears a little here so please feel free
>> to
>>>> correct this assumption.
>>>> * Is Hadoop really only good for jobs where the data doesn't fit on a
>>>> single node?  At some level, I assume that it can still speedup jobs
>> that do
>>>> fit on one node, if only because you are performing tasks in parallel.
>>>> 
>>>> Thanks!
>>>> 
>>>> --Andrew
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>> 
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


Mime
View raw message