hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suma Shivaprasad <sumasai.shivapra...@gmail.com>
Subject OOM with Hive on Tez
Date Tue, 26 Aug 2014 14:17:40 GMT
Trying to run a query on Tez with the following configurations


*set hive.tez.container.size=5120*
*set mapreduce.map.child.java.opts=-Xmx5120M*
*set hive.tez.java.opts=-Xmx4096M*
*set hive.auto.convert.join.noconditionaltask.size=805306000*
*set tez.am.resource.memory.mb=5120*
*set tez.am.java.opts=-Xmx4096M*

The above config settings were set after  running
https://github.com/hortonworks/hdp-configuration-utils/blob/master/2.1/hdp-configuration-utils.py
to get the right memory configs

Tried with both

set tez.runtime.io.sort.mb=512
set mapreduce.task.io.sort.mb=512

and

set tez.runtime.io.sort.mb=2048
set mapreduce.task.io.sort.mb=2048


The query I am trying run is

*select sum(tab1.m1),sum(tab1.m2)*
* from tab1 join tab2 dm on tab1.col1=tab2.col1*
* where tab1.dt = '2014-06-01' *
* and tab2.col2 = '..'*
* and tab2.col3 IN ('..')*
* group by TAB1.col1*

*where TAB1.col1 has high cardinality(around 700- 800 million)*

And its going OOM during shuffle phase.

 errorMessage=Fetch failed
Container released by application,
AttemptID:attempt_1407396011310_1577_1_01_000000_4 Info:Error:
exceptionThrown=java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
at
org.apache.tez.runtime.library.shuffle.common.MemoryFetchedInput.<init>(MemoryFetchedInput.java:38)
at
org.apache.tez.runtime.library.shuffle.common.impl.SimpleFetchedInputAllocator.allocate(SimpleFetchedInputAllocator.java:137)
at
org.apache.tez.runtime.library.shuffle.common.Fetcher.fetchInputs(Fetcher.java:252)
at
org.apache.tez.runtime.library.shuffle.common.Fetcher.call(Fetcher.java:184)
at
org.apache.tez.runtime.library.shuffle.common.Fetcher.call(Fetcher.java:59)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


Please advice if the configurations look ok? Do I need to change anything?



Thanks
Suma

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message