phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers
Date Wed, 23 May 2018 19:37:01 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487915#comment-16487915
] 

Hudson commented on PHOENIX-4685:
---------------------------------

FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1885 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1885/])
PHOENIX-4685 Properly handle connection caching for Phoenix inside (ankitsinghal59: rev 4c918352d1893bba46db2bdf08f468ca52fe2cba)
* (edit) phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java


> Properly handle connection caching for Phoenix inside RegionServers
> -------------------------------------------------------------------
>
>                 Key: PHOENIX-4685
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4685
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Rajeshbabu Chintaguntla
>            Assignee: Rajeshbabu Chintaguntla
>            Priority: Blocker
>             Fix For: 4.14.0, 5.0.0
>
>         Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch, PHOENIX-4685_addendum.patch,
PHOENIX-4685_addendum2.patch, PHOENIX-4685_addendum3.patch, PHOENIX-4685_addendum4.patch,
PHOENIX-4685_jstack, PHOENIX-4685_v2.patch, PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch,
PHOENIX-4685_v5.patch
>
>
> Currently trying to write data to indexed table failing with OOME where unable to create
native threads. But it's working fine with 4.7.x branches. Found many threads created for
meta lookup and shared threads and no space to create threads. This is happening even with
short circuit writes enabled.
> {noformat}
> 2018-04-08 13:06:04,747 WARN  [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020]
index.PhoenixIndexFailurePolicy: handleFailure failed
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
>         at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
>         at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
>         at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
>         at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
>         at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
>         at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
>         at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
>         at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
>         at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>         at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
>         at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
>         at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>         at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.reflect.UndeclaredThrowableException
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
>         at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
>         at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
>         at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40)
>         at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:183)
>          ... 25 more
> Caused by: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native
thread
>         at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:266)
>         at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
>         ... 32 more
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:714)
>         at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
>         at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
>         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1007)
>         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:986)
>         at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:724)
>         at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:709)
>         at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:236)
>         ... 36 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message