hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Goel <gan...@yahoo-inc.com>
Subject Re: Concatenating files on HDFS
Date Thu, 27 Aug 2009 05:34:30 GMT
HDFS files are write once so you cannot append to them (at the moment).
What you can do is copy your local file to HDFS dir containing the same file you want to append
Once that is done you can run a simple (Identity Mapper & Identity Reducer) mapreduce
job with input
as this directory and number of reducers = 1. 

----- Original Message -----
From: "Turner Kunkel" <thkunkel@gmail.com>
To: core-user@hadoop.apache.org
Sent: Wednesday, August 26, 2009 10:02:41 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
Subject: Concatenating files on HDFS

Is there any way to concatenate/append a local file to a file on HDFS
without copying down the HDFS file locally first?

I tried:
bin/hadoop dfs -cat file:///[local file] >> hdfs://[hdfs file]
But it just tries to look for hdfs://[hdfs file] as a local file,
since I suppose the dfs -cat command doesn't support the >> operator.


-Turner Kunkel

View raw message