hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dileep kumar <dileep...@gmail.com>
Subject Re: merging small files in HDFS
Date Fri, 04 Nov 2016 04:27:18 GMT
Hi ,

You need to write a map method to just parse input file and pass it to
reducer.. use only reducer..so that all maps output will go to one reducer
and one file gets created,which is merge of input files..

On 03-Nov-2016 8:54 pm, "Piyush Mukati" <piyush.mukati@gmail.com> wrote:

> Hi,
> I want to merge multiple files in one HDFS dir to one file. I am planning
> to write a map only job using input format which will create only one
> inputSplit per dir.
> this way my job don't need to do any shuffle/sort.(only read and write
> back to disk)
> Is there any such file format already implemented ?
> Or any there better solution for the problem.
> thanks.

View raw message