From dev-return-50792-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Wed Apr 11 16:36:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5754B18064A for ; Wed, 11 Apr 2018 16:36:05 +0200 (CEST) Received: (qmail 91050 invoked by uid 500); 11 Apr 2018 14:36:04 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 90993 invoked by uid 99); 11 Apr 2018 14:36:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Apr 2018 14:36:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id DF909C012B for ; Wed, 11 Apr 2018 14:36:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 1UOxxO-3fiQp for ; Wed, 11 Apr 2018 14:36:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id D461560D63 for ; Wed, 11 Apr 2018 14:36:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 08794E0BEE for ; Wed, 11 Apr 2018 14:36:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 63311241CB for ; Wed, 11 Apr 2018 14:36:00 +0000 (UTC) Date: Wed, 11 Apr 2018 14:36:00 +0000 (UTC) From: "Ankit Singhal (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-4685) Parallel writes continuously to indexed table failing with OOME very quickly in 5.x branch MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434008#comment-16434008 ] Ankit Singhal commented on PHOENIX-4685: ---------------------------------------- bq. Need to see does this have any side effects if we use the shared connection for all the operations. yeah, I think there could be following side effects:- * the whole purpose of using CoprocessorHConnectionTableFactory is to build the connection with custom configuration. if we use the shared connection then our index configuration will not take effect like our ServerRpcController , threads etc . see HBASE-18359 and PHOENIX-3974. * default value of 256 for "hbase.hconnection.threads.max" may be less for all index writers on the regionserver but in general scenario 8 threads per core(with common 32 core server) should be sufficient but not sure such limit result in any regression from previous versions or require documentation change. So , how about getting our own ShortCircuitConnection from the environment with custom config and pool , and shared the same across all region while writing? > Parallel writes continuously to indexed table failing with OOME very quickly in 5.x branch > ------------------------------------------------------------------------------------------ > > Key: PHOENIX-4685 > URL: https://issues.apache.org/jira/browse/PHOENIX-4685 > Project: Phoenix > Issue Type: Bug > Reporter: Rajeshbabu Chintaguntla > Assignee: Rajeshbabu Chintaguntla > Priority: Major > Fix For: 5.0.0 > > Attachments: PHOENIX-4685.patch, PHOENIX-4685_jstack > > > Currently trying to write data to indexed table failing with OOME where unable to create native threads. But it's working fine with 4.7.x branches. Found many threads created for meta lookup and shared threads and no space to create threads. This is happening even with short circuit writes enabled. > {noformat} > 2018-04-08 13:06:04,747 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020] index.PhoenixIndexFailurePolicy: handleFailure failed > java.io.IOException: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144) > at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632) > at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607) > at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533) > at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666) > at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) > Caused by: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761) > at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) > at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40) > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:183) > ... 25 more > Caused by: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native thread > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:266) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) > ... 32 more > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:714) > at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) > at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1007) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:986) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:724) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:709) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:236) > ... 36 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)