hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eltonsky <eltont9...@hotmail.com>
Subject What it's gonna happend when it comes to large number of maps?
Date Thu, 15 Apr 2010 10:49:48 GMT

Hello everyone,

I know when map func generates intermediate output, reduce func will pull
data directly from all maps' local disk. Although we can use combiner func
to minimize the amount of data, when we have many mappers, say 10,000, that
will be a crazy IO headache. And that dosen't seem right. 

Can anyone highlighten me on this?

Regards,
Elton
-- 
View this message in context: http://old.nabble.com/What-it%27s-gonna-happend-when-it-comes-to-large-number-of-maps--tp28253798p28253798.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.


Mime
View raw message