hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Premal Shah <premal.j.s...@gmail.com>
Subject Re: Hive/Tez local mode running out of memory
Date Sun, 30 Oct 2016 05:37:56 GMT
Hi Prasanth,
Thanx for the reply. We don't override the log level. According to the
docs, looks like the default level is INFO.
Any other ideas?

On Sat, Oct 29, 2016 at 3:25 PM, Prasanth Jayachandran <
pjayachandran@hortonworks.com> wrote:

> What is the value of hive.tez.log.level? My guess is this happens only in
> DEBUG log level.
>
> Thanks
> Prasanth
>
>
>
>
> On Fri, Oct 28, 2016 at 9:40 PM -0700, "Premal Shah" <
> premal.j.shah@gmail.com> wrote:
>
> Hive 2.0.1
> Hadoop 2.7.2
> Tex 0.8.4
>
> We have a UDF in hive which take in some values and outputs a score. When
> running a query on a table which calls the score function on every row,
> looks like tez is not running the query on YARN, but trying to run it in
> local mode. It then runs out of memory trying to insert that data into a
> table.
>
> Here's the query
>
>
> *ADD JAR score.jar; *
> *CREATE TEMPORARY FUNCTION score AS 'hive.udf.ScoreUDF';*
>
> *CREATE TABLE abc AS*
> *SELECT*
> *    id,*
> *    score(col1, col2) as score*
> *    , '2016-10-11' AS dt*
> *FROM input_table*
> *;*
>
> Here's the output of the shell
>
> Query ID = hadoop_20161028232841_5a06db96-ffaa-4e75-a657-c7cb46ccb3f5
> Total jobs = 1
> Launching Job 1 out of 1
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:3332)
>         at java.lang.AbstractStringBuilder.expandCapacity(
> AbstractStringBuilder.java:137)
>         at java.lang.AbstractStringBuilder.ensureCapacityInternal(
> AbstractStringBuilder.java:121)
>         at java.lang.AbstractStringBuilder.append(
> AbstractStringBuilder.java:622)
>         at java.lang.StringBuilder.append(StringBuilder.java:202)
>         at com.google.protobuf.TextFormat.escapeBytes(
> TextFormat.java:1283)
>         at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:394)
>         at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:283)
>         at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:283)
>         at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.access$400(
> TextFormat.java:248)
>         at com.google.protobuf.TextFormat.shortDebugString(
> TextFormat.java:88)
> FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask.
> Java heap space
>
>
> It looks like the job is not getting submitted to the cluster, but running
> locally. We can't get tez to run the query on the cluster.
> The hive shell starts with an Xmx of 4G.
>
> If I set hive.execution.engine = mr, then the query works, because it runs
> on the hadoop cluster.
>
> What should we change to avoid this problem?
>
> Thanx
>
> --
> Regards,
> Premal Shah.
>



-- 
Regards,
Premal Shah.

Mime
View raw message