hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Tatarinov <i...@decide.com>
Subject Re: OutOfMemory errors on joining 2 large tables.
Date Thu, 24 Feb 2011 06:12:35 GMT
The one setting that's still unknown
is mapred.tasktracker.reduce.tasks.maximum
Have you tried setting that to 1 as an experiment and increasing the java
heap.
I assume the mappers are not eating all the memory.



On Wed, Feb 23, 2011 at 9:57 PM, hadoop n00b <new2hive@gmail.com> wrote:

> I have suddenly began to get this error (hadoop error code 2) even for
> not-so-big queries. I am running a 6 node cluster. I tried to run the
> queries with 6 and 10 reducers but got the same result.
>
> On Wed, Feb 23, 2011 at 8:25 PM, Bennie Schut <bschut@ebuddy.com> wrote:
>
>> We filter nulls already before the tables are filled but then this will
>> probably cause a skew in the keys like Paul was saying. I'm running some
>> queries on the keys to see if that's the case.
>> I do expect there will be large differences in distribution of some of the
>> keys.
>> I'm looking at "set hive.optimize.skewjoin=true" before the query to see
>> if that helps. Will try that later.
>>
>>
>> On 02/23/2011 05:25 AM, Mapred Learn wrote:
>>
>>> Oops I meant nulls.
>>>
>>> Sent from my iPhone
>>>
>>> On Feb 22, 2011, at 8:22 PM, Mapred Learn<mapred.learn@gmail.com>
>>>  wrote:
>>>
>>> Check if you can filter non-nulls. That might help.
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Feb 22, 2011, at 12:46 AM, Bennie Schut<bschut@ebuddy.com>  wrote:
>>>>
>>>> I've just set the "hive.exec.reducers.bytes.per.reducer" to as low as
>>>>> 100k which caused this job to run with 999 reducers. I still have 5 tasks
>>>>> failing with an outofmemory.
>>>>>
>>>>> We have jvm reuse set to 8 but dropping it to 1 seems to greatly reduce
>>>>> this problem:
>>>>> set mapred.job.reuse.jvm.num.tasks = 1;
>>>>>
>>>>> It's still puzzling me how it can run out of memory. It seems like some
>>>>> of the reducers get an unequally large share of the work.
>>>>>
>>>>>
>>>>> On 02/18/2011 10:53 AM, Bennie Schut wrote:
>>>>>
>>>>>> When we try to join two large tables some of the reducers stop with
an
>>>>>> OutOfMemory exception.
>>>>>>
>>>>>> Error: java.lang.OutOfMemoryError: Java heap space
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1508)
>>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408)
>>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261)
>>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195)
>>>>>>
>>>>>>
>>>>>>
>>>>>> When looking at garbage collection for these reduce tasks it's
>>>>>> continually doing garbage collections.
>>>>>> Like this:
>>>>>> 2011-02-17T14:36:08.295+0100: 1250.547: [Full GC [PSYoungGen:
>>>>>> 111055K->53659K(233024K)] [ParOldGen: 698410K->698410K(699072K)]
>>>>>> 809466K->752070K(932096K) [PSPermGen: 14450K->14450K(21248K)],
>>>>>> 0.1496600
>>>>>> secs] [Times: user=1.08 sys=0.00, real=0.15 secs]
>>>>>> 2011-02-17T14:36:08.600+0100: 1250.851: [Full GC [PSYoungGen:
>>>>>> 111057K->53660K(233024K)] [ParOldGen: 698410K->698410K(699072K)]
>>>>>> 809468K->752070K(932096K) [PSPermGen: 14450K->14450K(21248K)],
>>>>>> 0.1360010
>>>>>> secs] [Times: user=1.00 sys=0.01, real=0.13 secs]
>>>>>> 2011-02-17T14:36:08.915+0100: 1251.167: [Full GC [PSYoungGen:
>>>>>> 111058K->53659K(233024K)] [ParOldGen: 698410K->698410K(699072K)]
>>>>>> 809468K->752070K(932096K) [PSPermGen: 14450K->14450K(21248K)],
>>>>>> 0.1325960
>>>>>> secs] [Times: user=0.94 sys=0.00, real=0.14 secs]
>>>>>> 2011-02-17T14:36:09.205+0100: 1251.457: [Full GC [PSYoungGen:
>>>>>> 111055K->53659K(233024K)] [ParOldGen: 698410K->698410K(699072K)]
>>>>>> 809466K->752070K(932096K) [PSPermGen: 14450K->14450K(21248K)],
>>>>>> 0.1301610
>>>>>> secs] [Times: user=0.99 sys=0.00, real=0.13 secs]
>>>>>>
>>>>>>
>>>>>> “mapred.child.java.opts” set to “-Xmx1024M -XX:+UseCompressedOops
>>>>>> -XX:+UseParallelOldGC -XX:+UseNUMA -Djava.net.preferIPv4Stack=true
>>>>>> -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCDetails
>>>>>> -Xloggc:/opt/hadoop/logs/task_@taskid@.gc.log”
>>>>>>
>>>>>> I've been reducing this parameter
>>>>>> “hive.exec.reducers.bytes.per.reducer”
>>>>>> to as low as 200M but I still get the OutOfMemory errors. I would
have
>>>>>> expected this would drop the amount of data send to the reducers
and
>>>>>> thus not have the OutOfMemory errors to happen.
>>>>>>
>>>>>> Any idea's on why this happens?
>>>>>>
>>>>>> I'm using a trunk build from around 2011-02-03
>>>>>>
>>>>>
>>
>

Mime
View raw message