phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chinmay Kulkarni (Jira)" <j...@apache.org>
Subject [jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
Date Sat, 28 Mar 2020 01:36:00 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chinmay Kulkarni updated PHOENIX-5802:
--------------------------------------
    Attachment: PHOENIX-5802-4.x-v2.patch

> Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory
iterator not being closed
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-5802
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5802
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 5.0.0, 4.15.0
>            Reporter: Chinmay Kulkarni
>            Assignee: Chinmay Kulkarni
>            Priority: Major
>             Fix For: 5.1.0, 4.16.0
>
>         Attachments: PHOENIX-5802-4.x-v1.patch, PHOENIX-5802-4.x-v2.patch
>
>          Time Spent: 3h
>  Remaining Estimate: 0h
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing Phoenix connection
and call [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].

>  Later we attempt to block on getting the result (iterator) of already-started tasks
and [close each iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had thrown an exception
_E_, we wouldn't be able to close the underlying iterator (which in-turn would'veĀ [closed
the cloned connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel", <exception E stack trace>
> {code}
> This is leading to a connection leak. Because we use CQSI to create the Phoenix Connection
here, it also gets accounted for during open Phoenix connection throttling, thus leaving the
client application with less connections that can be opened.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message