hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Dong <simond...@gmail.com>
Subject Re: Map-Reduce: How to make MR output one file an hour?
Date Sun, 02 Mar 2014 05:14:56 GMT
Fengyun,

Is there any particular reason you have to have exactly 1 file per hour? As
you probably knew already, each reducer will output 1 file, or if you use
MultipleOutputs as I suggested, a set of files. If you have to fit the
number of reducers to the number hours you have from the input, and
generate the number of files accordingly, it will most likely be at the
expense of cluster efficiency and performance. A worst case scenario of
course is if you have a bunch of data all within the same hour, then you
have to settle with 1 reducer without any parallelization at all.

A workaround is to use MultipleOutputs to generate a set of files for each
hour, with the hour being a the base name. Or if you so choose, a
sub-directory for each hour. For example if you use mmddhh as the base
name, you will have a set of files for an hour like:

030119-r-00000
...
030119-r-0000n
030120-r-00000
...
030120-r-0000n

Or in a sub-directory:

030119/part-r-00000
...
030119/part-r-0000n

You can then use wild card to glob the output either for manual processing,
or as input path for subsequent jobs.

-Simon



On Sat, Mar 1, 2014 at 7:37 PM, Fengyun RAO <raofengyun@gmail.com> wrote:

> Thanks Devin. We don't just want one file. It's complicated.
>
> if the input folder contains data in X hours, we want X files,
> if Y hours, we want Y files.
>
> obviously, X or Y is unknown on compile time.
>
> 2014-03-01 20:48 GMT+08:00 Devin Suiter RDX <dsuiter@rdx.com>:
>
>> If you only want one file, then you need to set the number of reducers to
>> 1.
>>
>> If the size of the data makes the original MR job impractical to use a
>> single reducer, you run a second job on the output of the first, with the
>> default mapper and reducer, which are the Identity- ones, and set that
>> numReducers = 1.
>>
>> Or use hdfs getmerge function to collate the results to one file.
>> On Mar 1, 2014 4:59 AM, "Fengyun RAO" <raofengyun@gmail.com> wrote:
>>
>>> Thanks, but how to set reducer number to X? X is dependent on input
>>> (run-time), which is unknown on job configuration (compile time).
>>>
>>>
>>> 2014-03-01 17:44 GMT+08:00 AnilKumar B <akumarb2010@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> Write the custom partitioner on <timestamp> and as you mentioned set
>>>> #reducers to X.
>>>>
>>>>
>>>>
>>>
>

Mime
View raw message