Return-Path: X-Original-To: apmail-tajo-dev-archive@minotaur.apache.org Delivered-To: apmail-tajo-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1C87710C5B for ; Tue, 21 Jan 2014 02:15:46 +0000 (UTC) Received: (qmail 49158 invoked by uid 500); 21 Jan 2014 02:15:45 -0000 Delivered-To: apmail-tajo-dev-archive@tajo.apache.org Received: (qmail 49105 invoked by uid 500); 21 Jan 2014 02:15:44 -0000 Mailing-List: contact dev-help@tajo.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@tajo.incubator.apache.org Delivered-To: mailing list dev@tajo.incubator.apache.org Received: (qmail 49074 invoked by uid 99); 21 Jan 2014 02:15:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jan 2014 02:15:43 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 21 Jan 2014 02:15:41 +0000 Received: (qmail 48491 invoked by uid 99); 21 Jan 2014 02:15:21 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jan 2014 02:15:21 +0000 Date: Tue, 21 Jan 2014 02:15:21 +0000 (UTC) From: "Min Zhou (JIRA)" To: dev@tajo.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (TAJO-522) OutOfMemoryError: unable to create new native thread MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/TAJO-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877098#comment-13877098 ] Min Zhou commented on TAJO-522: ------------------------------- Hi all, I ran a big query applied this patch, still OOM. Please submit this patch and I will the next step. > OutOfMemoryError: unable to create new native thread > ---------------------------------------------------- > > Key: TAJO-522 > URL: https://issues.apache.org/jira/browse/TAJO-522 > Project: Tajo > Issue Type: Bug > Affects Versions: 0.8-incubating > Reporter: Min Zhou > Assignee: hyoungjunkim > Priority: Critical > Fix For: 0.8-incubating > > Attachments: TAJO-522.patch, tajo-site.xml > > > Another exception occurs on the client side when I run a query > {noformat} > Progress: 0%, response time: 190.197 sec > com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: unable to create new native thread > at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:193) > at org.apache.tajo.cli.TajoCli.getQueryResult(TajoCli.java:353) > at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:319) > at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228) > at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735) > Caused by: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread > at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76) > at org.apache.tajo.rpc.BlockingRpcClient.(BlockingRpcClient.java:71) > at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54) > at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63) > at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:188) > ... 4 more > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:640) > at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657) > at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38) > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343) > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:95) > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:53) > at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28) > at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99) > at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33) > at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:151) > at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:116) > at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54) > ... 8 more > java.lang.OutOfMemoryError: unable to create new native thread > 2014-01-19 23:12:19,975 WARN client.TajoClient (TajoClient.java:closeQuery(110)) - Fail to close a QueryMaster connection (qid=q_1390100273039_0012, msg=java.lang.OutOfMemoryError: unable to create new native thread) > java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread > at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76) > at org.apache.tajo.rpc.BlockingRpcClient.(BlockingRpcClient.java:71) > at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54) > at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63) > at org.apache.tajo.client.TajoClient.closeQuery(TajoClient.java:106) > at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:323) > at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228) > at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:640) > at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657) > at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38) > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343) > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:95) > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:53) > at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28) > at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99) > at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39) > at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33) > at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:151) > at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:116) > at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54) > ... 7 more > {noformat} > I am not sure why client side need to create quite a lot of threads. -- This message was sent by Atlassian JIRA (v6.1.5#6160)