From dev-return-60616-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Wed Apr 1 20:30:03 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id E5AD4180660 for ; Wed, 1 Apr 2020 22:30:02 +0200 (CEST) Received: (qmail 72288 invoked by uid 500); 1 Apr 2020 20:30:01 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 72270 invoked by uid 99); 1 Apr 2020 20:30:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2020 20:30:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id A5CA5E2C44 for ; Wed, 1 Apr 2020 20:30:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 2689F78028F for ; Wed, 1 Apr 2020 20:30:00 +0000 (UTC) Date: Wed, 1 Apr 2020 20:30:00 +0000 (UTC) From: "Chinmay Kulkarni (Jira)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-5802?page=3Dcom.atlass= ian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5802: -------------------------------------- Affects Version/s: 4.14.3 > Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIte= ratorFactory iterator not being closed > -------------------------------------------------------------------------= -------------------------------------- > > Key: PHOENIX-5802 > URL: https://issues.apache.org/jira/browse/PHOENIX-5802 > Project: Phoenix > Issue Type: Bug > Affects Versions: 5.0.0, 4.15.0, 4.14.3 > Reporter: Chinmay Kulkarni > Assignee: Chinmay Kulkarni > Priority: Major > Labels: phoenix-hardening > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5802-4.x-v1.patch, PHOENIX-5802-4.x-v2.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > During an UPSERT SELECT query or client-side DELETE, we clone the existin= g Phoenix connection and call [mutate|https://github.com/apache/phoenix/blo= b/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/a= pache/phoenix/compile/MutatingParallelIteratorFactory.java#L59] which event= ually calls either [upsertSelect|https://github.com/apache/phoenix/blob/73a= 86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache= /phoenix/compile/UpsertCompiler.java#L178] or [deleteRows|https://github.co= m/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core= /src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126]. > When getting iterators, if we encounter any exception, then on calling [c= lose|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de412392= 11330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultI= terators.java#L1377], we attempt to cancel any queued work, while [accumula= ting already-started task futures|https://github.com/apache/phoenix/blob/73= a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apach= e/phoenix/iterate/BaseResultIterators.java#L1453].=20 > Later we attempt to block on getting the result (iterator) of already-st= arted tasks and [close each iterator|https://github.com/apache/phoenix/blob= /73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/ap= ache/phoenix/iterate/BaseResultIterators.java#L1464-L1465]. > Here however, if any of the upsert-select or client-side delete tasks had= thrown an exception _E_, we wouldn't be able to close the underlying itera= tor (which in-turn would've=C2=A0[closed the cloned connection|https://gith= ub.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix= -core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFact= ory.java#L101]). This manifests as [these logs|https://github.com/apache/ph= oenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/j= ava/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]: > {code:java} > "Failed to execute task during cancel", > {code} > This is leading to a connection leak. Because we use CQSI to create the P= hoenix Connection here, it also gets accounted for during open Phoenix conn= ection throttling, thus leaving the client application with less connection= s that can be opened. -- This message was sent by Atlassian Jira (v8.3.4#803005)