From dev-return-51736-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Sat May 12 01:25:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D876A180647 for ; Sat, 12 May 2018 01:25:04 +0200 (CEST) Received: (qmail 54599 invoked by uid 500); 11 May 2018 23:25:03 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 54588 invoked by uid 99); 11 May 2018 23:25:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 May 2018 23:25:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 7067FC887B for ; Fri, 11 May 2018 23:25:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id WIcT6hL8pNoj for ; Fri, 11 May 2018 23:25:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id D35B05F4E5 for ; Fri, 11 May 2018 23:25:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id CF175E0F88 for ; Fri, 11 May 2018 23:25:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 29E5F21548 for ; Fri, 11 May 2018 23:25:00 +0000 (UTC) Date: Fri, 11 May 2018 23:25:00 +0000 (UTC) From: "Hudson (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472794#comment-16472794 ] Hudson commented on PHOENIX-4685: --------------------------------- ABORTED: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #126 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/126/]) PHOENIX-4685 Properly handle connection caching for Phoenix inside (rajeshbabu: rev 39b92bf9e8d9cae46b1fa230d91ac04a8e49e629) * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java > Properly handle connection caching for Phoenix inside RegionServers > ------------------------------------------------------------------- > > Key: PHOENIX-4685 > URL: https://issues.apache.org/jira/browse/PHOENIX-4685 > Project: Phoenix > Issue Type: Bug > Reporter: Rajeshbabu Chintaguntla > Assignee: Rajeshbabu Chintaguntla > Priority: Blocker > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch, PHOENIX-4685_addendum.patch, PHOENIX-4685_jstack, PHOENIX-4685_v2.patch, PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch, PHOENIX-4685_v5.patch > > > Currently trying to write data to indexed table failing with OOME where unable to create native threads. But it's working fine with 4.7.x branches. Found many threads created for meta lookup and shared threads and no space to create threads. This is happening even with short circuit writes enabled. > {noformat} > 2018-04-08 13:06:04,747 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020] index.PhoenixIndexFailurePolicy: handleFailure failed > java.io.IOException: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144) > at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632) > at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607) > at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533) > at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666) > at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) > Caused by: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761) > at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) > at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40) > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:183) > ... 25 more > Caused by: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native thread > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:266) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) > ... 32 more > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:714) > at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) > at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1007) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:986) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:724) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:709) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:236) > ... 36 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)