phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas D'Silva (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-2690) Exceeding OS limit for native threads with large number of guide posts
Date Thu, 18 Feb 2016 21:11:18 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153091#comment-15153091
] 

Thomas D'Silva commented on PHOENIX-2690:
-----------------------------------------

[~mujtabachohan] When I run the query on a table with a large number of guideposts (>4000)
it crashes my local HBase but I wasn't able to repro the OutOfMemoryError. 

After I restart my local HBase any query on the table fail with the following exception

{code}
Error: Task org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask@df76c5e rejected from
org.apache.phoenix.job.JobManager$1@51f722f1[Running, pool size = 128, active threads = 128,
queued tasks = 5000, completed tasks = 5168] (state=08000,code=101)
{code}

 BaseQueryServicesImpl seems to be setting up the executor correctly. 
{code}
this.executor =  JobManager.createThreadPoolExec(
                options.getKeepAliveMs(), 
                options.getThreadPoolSize(), 
                options.getQueueSize(),
                options.isGlobalMetricsEnabled());
{code}

Can you run ulimit -a on one of the datanodes to see what the process limit is set to? Was
anything else running while you ran your query?

> Exceeding OS limit for native threads with large number of guide posts
> ----------------------------------------------------------------------
>
>                 Key: PHOENIX-2690
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2690
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.7.0
>            Reporter: Mujtaba Chohan
>            Assignee: Thomas D'Silva
>            Priority: Minor
>
> With >32K guidepost count for a table (which equates to a 320 GB table size with 10MB
guidepost width), following OOM occurs on 64bit Linux client
> {code}
> select * from table_with_32k_guideposts limit 1000;
> java.lang.OutOfMemoryError: unable to create new native thread
> 	at java.lang.Thread.start0(Native Method)
> 	at java.lang.Thread.start(Thread.java:714)
> 	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) at org.apache.phoenix.job.JobManager$InstrumentedThreadPoolExecutor.execute(JobManager.java:261)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) at
org.apache.phoenix.iterate.ParallelIterators.submitWork(ParallelIterators.java:103) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:621)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:591)
at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message