hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: Hadoop Streaming -file option
Date Mon, 23 Feb 2009 18:29:35 GMT

On Feb 23, 2009, at 2:01 AM, Bing TANG wrote:

> Hi, everyone,
> Could somdone tell me the principle of "-file" when using Hadoop
> Streaming. I want to ship a big file to Slaves, so how it works?
>
> Hadoop uses "SCP" to copy? How does Hadoop deal with -file option?
>

No, -file just copies the file from the local filesystem to HDFS, and  
the DistributedCache copies it to the local filesystem of the node on  
which the map/reduce task runs.

Arun


Mime
View raw message