crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Ortiz <dpo5...@gmail.com>
Subject Re: temporary table size is 0, which makes reducer number too small
Date Mon, 17 Oct 2016 03:19:31 GMT
That gets tricky if you have input data that is heavily filtered though.
Perhaps play around with the scale factor on operations that may blow up
data?

On Sun, Oct 16, 2016, 10:04 PM 陈竞 <cj.magina@gmail.com> wrote:

> that's  a solution, but, since user may not clearly know whic step will
> produce tempoary table, i think setting reduce number  automatically will
> improve user experience. I think maybe we can set reduce number as 1/3
> mapper number before submitting jobs if one of the job inputs is temporary
> table.
>
> 2016-10-14 18:59 GMT+08:00 David Ortiz <dpo5003@gmail.com>:
>
> You can manually set the reducer number using the conf object among other
> things.
>
> On Fri, Oct 14, 2016, 5:43 AM 陈竞 <cj.magina@gmail.com> wrote:
>
> hi, i found that if the pipeline produce temporary table , the reduce
> number of the temporary table whose input table is temporary table  become
> to small, since temporary table has no content .
>
>
>
>
> --
> 陈竞,中科院计算技术研究所,高性能计算机中心
> Jing Chen HPCC.ICT.AC China
>

Mime
View raw message