hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gautam <gautamkows...@gmail.com>
Subject Re: Record too large for Tez in-memory buffer...
Date Thu, 11 Feb 2016 03:46:00 GMT
Here's the json version.

On Wed, Feb 10, 2016 at 7:44 PM, Gautam <gautamkowshik@gmail.com> wrote:

> Whoops.. meant to send the tez explain earlier. Here's the Tez query plan.
> Good to know there's a fix .. Is there a jira that talks about this issue? Coz
> I couldn't find one. Maybe I can alter the query a bit to filter these
> out.
>
> Cheers,
> -Gautam.
>
>
> On Wed, Feb 10, 2016 at 7:32 PM, Gopal Vijayaraghavan <gopalv@apache.org>
> wrote:
>
>> Hey,
>>
>> > Trying to benchmark with Hive on Tez causes the following error.
>> >Admittedly these are some very large looking records .. the same job runs
>> >fine on MR2.
>> ...
>> > I'v attached the query explain tree.  It fails in the very last reducer
>> >phase ..
>>
>> Can you attach the explain plan with hive.execution.engine=tez (even
>> better would be the JSON output of "explain formatted").
>>
>> > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>>
>> >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo
>> >oSmallException: Record too large for in-memory buffer. Exceeded buffer
>> >overflow limit, bufferOverflowRecursion=2, bufferList.size=1,
>> >blockSize=268435456
>>
>> You have a single row which is > 200Mb?
>>
>> The tez 0.8.2 out there should already have this issue fixed, but single
>> spill record mode is likely to spin your disks all wrong (unless you have
>> SSDs).
>>
>> I'm guessing there's a UDTF + PTF producing humongous rows & that it's
>> being planned wrong because that's not a standard pattern.
>>
>> Cheers,
>> Gopal
>>
>>
>>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>



-- 
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."

Mime
View raw message