hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wataru Yukawa (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HIVE-13274) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on MapReduce
Date Sun, 13 Mar 2016 12:07:33 GMT

     [ https://issues.apache.org/jira/browse/HIVE-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wataru Yukawa resolved HIVE-13274.
----------------------------------
    Resolution: Duplicate

> "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on MapReduce
> --------------------------------------------------------------------------------------------
>
>                 Key: HIVE-13274
>                 URL: https://issues.apache.org/jira/browse/HIVE-13274
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>         Environment: HDP2.3.4
> JDK1.8
> Centos6
>            Reporter: Wataru Yukawa
>            Assignee: Vaibhav Gumashta
>
> "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000
> HiveServer2 memory leak occurs.
> Heapstats result is https://gyazo.com/27dcaf678fb8d2e4003af55a79c2020e
> hiveserver2.log
> {code}
> 2016-03-13 13:25:06,041 INFO  [HiveServer2-Handler-Pool: Thread-98838]: retry.RetryInvocationHandler
(RetryInvocationHandler.java:invoke(144)) - Exception while invoking getFileInfo of class
Cl
> ientNamenodeProtocolTranslatorPB over ...:8020 after 3 fail over attempts. Trying to
fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up
IO streams; Host Details : local host is: "..."; destination host is: "...":8020; 
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1431)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>         at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>         at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1315)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>         at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:85)
>         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:95)
>         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:190)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
>         at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
>         at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
>         at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
>         at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1096)
>         at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
>         at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183)
>         at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
>         at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261)
>         at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
>         at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
>         at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
>         at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Couldn't set up IO streams
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:791)
>         at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1397)
>         ... 57 more
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:714)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:784)
>         ... 60 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message