tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Min Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-525) Concurrent queries hang
Date Mon, 20 Jan 2014 02:53:19 GMT

    [ https://issues.apache.org/jira/browse/TAJO-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876106#comment-13876106
] 

Min Zhou commented on TAJO-525:
-------------------------------

The internal error is caused by a failure of an execution block. I changed some code, and
found something similar to TAJO-522 

{noformat}
014-01-20 01:53:04,959 ERROR master.ContainerProxy (TajoContainerProxy.java:assignExecutionBlock(88))
- java.lang.OutOfMemoryError: unable to create new native thread
java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
        at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
        at org.apache.tajo.rpc.AsyncRpcClient.<init>(AsyncRpcClient.java:67)
        at org.apache.tajo.master.TajoContainerProxy.assignExecutionBlock(TajoContainerProxy.java:72)
        at org.apache.tajo.master.TajoContainerProxy.launch(TajoContainerProxy.java:59)
        at org.apache.tajo.worker.TajoResourceAllocator$LaunchRunner.run(TajoResourceAllocator.java:189)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
        at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
       at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
        at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
        at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
        at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
        ... 10 more
{noformat}

This process ate a lot of memory for threads, I even can't launch jstack on that machine

{noformat}
$ jps
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /export/home/yarn/hs_err_pid19677.log
{noformat}

pstack shows the process started over 1000 threads, will submit the pstack result soon.





> Concurrent queries hang
> -----------------------
>
>                 Key: TAJO-525
>                 URL: https://issues.apache.org/jira/browse/TAJO-525
>             Project: Tajo
>          Issue Type: Bug
>    Affects Versions: 0.8-incubating
>            Reporter: Min Zhou
>            Priority: Critical
>             Fix For: 0.8-incubating
>
>
> When I run a simple query "select count(1) from lineitem;" in one tsql session and after
that run the 2nd query in another tsql session.  The 2nd one hung.  
> After the 1st one done,  the 2nd one still stuck at 0%.
> After that, run the same query in the first tsql session, the 3rd query also hung.  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message