hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Noguchi <knogu...@yahoo-inc.com>
Subject Re: Upload files directly to hdfs from standard out using pipes
Date Wed, 12 Jan 2011 16:11:31 GMT
1. Run map-only (no reducer) job.  Output goes directly to hdfs.
or
2. echo abc | hadoop dfs ­put - /user/knoguchi/somefile
   would write to hdfs reading from stdin.

Koji

On 1/12/11 2:07 AM, "Mapred Learn" <mapred.learn@gmail.com> wrote:

> Hi,
> I found out that :
> https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs/
> hadoopfs.py 
> <https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs
> /hadoopfs.py> 
> 
> can be used to write data directly to HDFS without writing to a local
> filesystem but I am not able to understand how.
> 
> Could somebody tell me how can I do this ? Or is there some other best way to
> do it ?
> 
> Basiclly my use case is to pipe standard out from a program directly
> to HDFS so that it does not have to go through the disk write.
> 
> 
> Thanks in advance !
> 
> 


Mime
View raw message