hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <ar...@yahoo-inc.com>
Subject Re: Reduce Performance
Date Sat, 18 Aug 2007 13:59:18 GMT
Thorsten,

On Sat, Aug 18, 2007 at 01:18:06PM +0200, Thorsten Schuett wrote:

>Please correct me if I am wrong, but it looks like hadoop is using the rpc
>mechanism to copy the map output files to the reduce task (in this case via
>the loopback device). i

Hadoop uses http to copy map output files, not rpc.

>If my assumptions are correct, would it be possible to
>read/access the files directly in the "one-node mode"?

Please take a look at LocalJobRunner in src/org/apache/hadoop/mapred ... set the jobtracker
in your config to 'local' and this happens automatically. (http://wiki.apache.org/lucene-hadoop/HowToDebugMapReducePrograms)

Arun

>Thanks,
>  Thorsten

Mime
View raw message