hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: Record too large for Tez in-memory buffer...
Date Thu, 11 Feb 2016 04:12:08 GMT

> Good to know there's a fix .. Is there a jira that talks about this
>issue? Coz I couldn't find one.

https://github.com/apache/tez/commit/714461f47e6408ec331acd0ddd640335e6a7a0
6c


Also, it looks like Reducer 16 is the one failing - not Reducer 17.

You can draw out the explain using https://github.com/t3rmin4t0r/lipwig

PTF doesn't actually tell the UDAF name in the explain, so I'm guessing it
a ROW_NUMBER() <= 50 - because that's the only one which didn't get
optimized.

I see absolutely no broadcast edges in this, so it's possible to disable
the weighted memory scaler in Tez to sort of dumb it down to MRv2 mode.

set tez.task.scale.memory.enabled=false;

*or* do extensive tuning for it (see
tez.task.scale.memory.additionalreservation.fraction.max).

Cheers,
Gopal


Mime
View raw message