hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Re: The dreaded Heap Space Issue on a Transform
Date Wed, 30 Jan 2013 14:28:29 GMT
So just a follow-up. I am less looking for specific troubleshooting on how
to fix my problem, and more looking for a general understanding of heap
space usage with Hive.  When I get an error like this, is it heap space on
a node, or heap space on my hive server?  Is it the heap space of the
tasktracker? Heap of the job kicked off on the node?  Which heap is being
affected? If it's not clear in my output, where can I better understand
this? I am sorely out of my league here when it comes to understanding the
JVM interactions of Hive and Hadoop, i.e. where hive is run, vs where task
trackers are run etc.

Thanks is advance!



On Tue, Jan 29, 2013 at 7:43 AM, John Omernik <john@omernik.com> wrote:

> I am running a transform script that parses through a bunch of binary
> data. In 99% of the cases it runs, it runs fine, but on certain files I get
> a failure (as seen below).  Funny thing is, I can run a job with "only" the
> problem source file, and it will work fine, but when as a group of files, I
> get these warnings.  I guess what I am asking here is this: Where is the
> heap error? Is this occurring on the nodes themselves or, since this is
> where the script is emitting records (and potentially large ones at that)
> and in this case my hive server running the job may be memory light, could
> the issue actually be due to heap on the hive server itself?   My setup is
> 1 Hive node (that is woefully underpowered, under memoried, and under disk
> I/Oed) and 4 beefy hadoop nodes.  I guess, my question is the heap issue on
> the sender or the receiver :)
>
>
>
>
> 13-01-29 08:20:24,107 INFO org.apache.hadoop.hive.ql.io.CodecPool: Got
> brand-new compressor
> 2013-01-29 08:20:24,107 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 1 rows
> 2013-01-29 08:20:24,410 INFO
> org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 10 rows
> 2013-01-29 08:20:24,410 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 10 rows
> 2013-01-29 08:20:24,411 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 10 rows
> 2013-01-29 08:20:24,411 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10 rows
> 2013-01-29 08:20:24,411 INFO
> org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 10 rows
> 2013-01-29 08:20:24,411 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 10 rows
> 2013-01-29 08:20:24,411 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 10 rows
> 2013-01-29 08:20:24,412 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 10 rows
> 2013-01-29 08:20:27,170 INFO
> org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 100 rows
> 2013-01-29 08:20:27,170 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 100 rows
> 2013-01-29 08:20:27,170 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 100 rows
> 2013-01-29 08:20:27,171 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows
> 2013-01-29 08:20:27,171 INFO
> org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 100 rows
> 2013-01-29 08:20:27,171 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 100 rows
> 2013-01-29 08:20:27,171 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 100 rows
> 2013-01-29 08:20:27,171 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 100 rows
> 2013-01-29 08:21:16,247 INFO
> org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 1000 rows
> 2013-01-29 08:21:16,247 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 1000 rows
> 2013-01-29 08:21:16,247 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 1000 rows
> 2013-01-29 08:21:16,247 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
> 2013-01-29 08:21:16,248 INFO
> org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 1000 rows
> 2013-01-29 08:21:16,248 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 1000 rows
> 2013-01-29 08:21:16,248 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 1000 rows
> 2013-01-29 08:21:16,248 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 1000 rows
> 2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 10000 rows
> 2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 10000 rows
> 2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 10000 rows
> 2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
>  2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 10000 rows
> 2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 10000 rows
> 2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 10000 rows
> 2013-01-29 08:25:47,532 INFO
> org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 10000 rows
> 2013-01-29 08:27:34,276 WARN
> org.apache.hadoop.hive.ql.exec.ScriptOperator: Exception in
> StreamThread.run(): Java heap space
> Cause: null
> 2013-01-29 08:27:34,277 WARN
> org.apache.hadoop.hive.ql.exec.ScriptOperator: java.lang.OutOfMemoryError:
> Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
>  at java.lang.String.<init>(String.java:215)
> at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:542)
>  at java.nio.CharBuffer.toString(CharBuffer.java:1157)
> at org.apache.hadoop.io.Text.decode(Text.java:350)
>  at org.apache.hadoop.io.Text.decode(Text.java:327)
> at org.apache.hadoop.io.Text.toString(Text.java:254)
>  at java.lang.String.valueOf(String.java:2826)
> at java.lang.StringBuilder.append(StringBuilder.java:115)
>  at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:873)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181)
>  at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76)
>  at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
>  at
> org.apache.hadoop.hive.ql.exec.ScriptOperator$OutputStreamProcessor.processLine(ScriptOperator.java:477)
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator$StreamThread.run(ScriptOperator.java:563)
>
> 2013-01-29 08:27:34,306 INFO
> org.apache.hadoop.hive.ql.exec.ScriptOperator: ErrorStreamProcessor calling
> reporter.progress()
> 2013-01-29 08:27:34,307 INFO
> org.apache.hadoop.hive.ql.exec.ScriptOperator: StreamThread ErrorProcessor
> done
> 2013-01-29 08:27:34,307 ERROR
> org.apache.hadoop.hive.ql.exec.ScriptOperator: Script failed with code 1
>

Mime
View raw message