hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Re: Append MR output file to an exitsted HDFS file
Date Mon, 22 Apr 2013 04:04:08 GMT
I might be misunderstanding, but if you want each Reducer to append its
outputs to outputs to corresponding files that already exist in HDFS?

Remember that the reducers usually are outputting globs so you will have
several parts to your output - so the append has to be done in a way where
new reducer paritions corresponds to a old paritions.

If so - maybe you could play with your own OutputFormat by taking the
source from one that serves as a starting point, and replacing the
"create..." stream part with a call to write() with a call to append().

The reason this is tricky is that each OutputFormat is going to have to
find the corresponding file to append.

On Sun, Apr 21, 2013 at 10:54 PM, YouPeng Yang <yypvsxf19870706@gmail.com>wrote:

> Hi All
>      Can I append a MR output file to an existed file on HDFS.
>      I‘m using CDH4.1.2 vs MRv2
> Regards

Jay Vyas

View raw message