hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gautam <gautamkows...@gmail.com>
Subject Re: Record too large for Tez in-memory buffer...
Date Fri, 12 Feb 2016 07:08:30 GMT
setting tez.task.scale.memory.enabled=false; failed the job sooner.

But the fix from TEZ-2575  worked!

Upgraded from 0.7.0 to 0.7.1 and applied
https://github.com/apache/tez/commit/0e155e7185d1350f64dead488103777295ac76d1


Goes through without any fatal issues. Will continue testing / benchmarking
further.

thanks!
-Gautam.

On Wed, Feb 10, 2016 at 8:12 PM, Gopal Vijayaraghavan <gopalv@apache.org>
wrote:

>
> > Good to know there's a fix .. Is there a jira that talks about this
> >issue? Coz I couldn't find one.
>
> https://github.com/apache/tez/commit/714461f47e6408ec331acd0ddd640335e6a7a0
> 6c
>
>
> Also, it looks like Reducer 16 is the one failing - not Reducer 17.
>
> You can draw out the explain using https://github.com/t3rmin4t0r/lipwig
>
> PTF doesn't actually tell the UDAF name in the explain, so I'm guessing it
> a ROW_NUMBER() <= 50 - because that's the only one which didn't get
> optimized.
>
> I see absolutely no broadcast edges in this, so it's possible to disable
> the weighted memory scaler in Tez to sort of dumb it down to MRv2 mode.
>
> set tez.task.scale.memory.enabled=false;
>
> *or* do extensive tuning for it (see
> tez.task.scale.memory.additionalreservation.fraction.max).
>
> Cheers,
> Gopal
>
>


-- 
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."

Mime
View raw message