hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Benson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11004) PermGen OOM error in Hiveserver2
Date Mon, 20 Jul 2015 11:50:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633464#comment-14633464
] 

Martin Benson commented on HIVE-11004:
--------------------------------------

Thanks Mostafa and apologies for the delayed reponse!

I tried using that setting but it didn't work. I think the ORC thing is perhaps a red herring
- there are other errors related to permgen space once HS2 is unresponsive. e.g.:

2015-07-20 09:37:25,114 WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error opening
session: 
org.apache.hive.service.cli.HiveSQLException: Failed to open new session: java.lang.RuntimeException:
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:256)
	at org.apache.hive.service.cli.CLIService.openSession(CLIService.java:191)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:405)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:297)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:472)
	at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:134)
	at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:254)
	... 12 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1488)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:64)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2841)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2860)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:453)
	... 14 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1486)
	... 19 more
Caused by: java.lang.OutOfMemoryError: PermGen space

In case it helps, this behaviour happens every time I try to run a particular, relatively
complex query. It is a chain of views with depends on each other that then come together in
a persisted table. e.g. along the lines of (but obviously more complicated!)

CREATE VIEW tmpvw1 as SELECT ... FROM sometable;
CREATE VIEW tmpvw2 as SELECT ... FROM tmpvw1;
CREATE VIEW tmpvw3 as SELECT ... FROM tmpvw2;
CREATE VIEW tmpvw4 as SELECT ... FROM tmpvw3;
CREATE VIEW tmpvw5 as SELECT ... FROM tmpvw4;
CREATE VIEW tmpvw6 as SELECT ... FROM tmpvw5;
CREATE VIEW tmpvw7 as SELECT ... FROM tmpvw6;
CREATE VIEW tmpvw8 as SELECT ... FROM tmpvw7;
CREATE VIEW tmpvw9 as SELECT ... FROM tmpvw8;
CREATE VIEW tmpvw10 as SELECT ... FROM tmpvw9;
CREATE TABLE newtable as SELECT... FROM tmpvw10;

> PermGen OOM error in Hiveserver2
> --------------------------------
>
>                 Key: HIVE-11004
>                 URL: https://issues.apache.org/jira/browse/HIVE-11004
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 1.1.0
>         Environment: cdh 5.4
>            Reporter: Martin Benson
>            Priority: Critical
>
> Periodically Hiveserver2 will become unresponsive and looking in the logs there is the
following error:
> {noformat}
> 2:28:22.965 PM	ERROR	org.apache.hadoop.hive.ql.io.orc.OrcInputFormat	
> Unexpected Exception
> java.lang.OutOfMemoryError: PermGen space
> 2:28:22.969 PM	WARN	org.apache.hive.service.cli.thrift.ThriftCLIService	
> Error fetching results: 
> org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException:
serious problem
> 	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:343)
> 	at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:250)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:656)
> 	at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451)
> 	at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: java.lang.RuntimeException: serious problem
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
> 	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:138)
> 	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1655)
> 	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:338)
> 	... 13 more
> Caused by: java.lang.RuntimeException: serious problem
> 	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$Context.waitForTasks(OrcInputFormat.java:478)
> 	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:944)
> 	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:969)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:362)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:294)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
> 	... 17 more
> Caused by: java.lang.OutOfMemoryError: PermGen space
> {noformat}
> There does not appear to be an obvious trigger for this (other than the fact that the
error mentions ORC). If further details would be helpful in diagnosing the issue please let
me know and I'll supply them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message