From dev-return-50767-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Tue Apr 10 16:12:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0949118064C for ; Tue, 10 Apr 2018 16:12:05 +0200 (CEST) Received: (qmail 48766 invoked by uid 500); 10 Apr 2018 14:12:05 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 48749 invoked by uid 99); 10 Apr 2018 14:12:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Apr 2018 14:12:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 92CA5C03DE for ; Tue, 10 Apr 2018 14:12:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id zS6oV2A0_SFM for ; Tue, 10 Apr 2018 14:12:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id A2AF15FAD9 for ; Tue, 10 Apr 2018 14:12:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 6A1EDE0110 for ; Tue, 10 Apr 2018 14:12:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id BB241241D6 for ; Tue, 10 Apr 2018 14:12:00 +0000 (UTC) Date: Tue, 10 Apr 2018 14:12:00 +0000 (UTC) From: "Rajeshbabu Chintaguntla (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (PHOENIX-4685) Parallel writes continuously to indexed table failing with OOME very quickly in 5.x branch MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla updated PHOENIX-4685: --------------------------------------------- Attachment: PHOENIX-4685.patch > Parallel writes continuously to indexed table failing with OOME very quickly in 5.x branch > ------------------------------------------------------------------------------------------ > > Key: PHOENIX-4685 > URL: https://issues.apache.org/jira/browse/PHOENIX-4685 > Project: Phoenix > Issue Type: Bug > Reporter: Rajeshbabu Chintaguntla > Assignee: Rajeshbabu Chintaguntla > Priority: Major > Fix For: 5.0.0 > > Attachments: PHOENIX-4685.patch, PHOENIX-4685_jstack > > > Currently trying to write data to indexed table failing with OOME where unable to create native threads. But it's working fine with 4.7.x branches. Found many threads created for meta lookup and shared threads and no space to create threads. This is happening even with short circuit writes enabled. > {noformat} > 2018-04-08 13:06:04,747 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020] index.PhoenixIndexFailurePolicy: handleFailure failed > java.io.IOException: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144) > at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632) > at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607) > at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533) > at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666) > at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) > Caused by: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761) > at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) > at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40) > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:183) > ... 25 more > Caused by: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native thread > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:266) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) > ... 32 more > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:714) > at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) > at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1007) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:986) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:724) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:709) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:236) > ... 36 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)