impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailesh Mukil (JIRA)" <>
Subject [jira] [Resolved] (IMPALA-4456) Change query_exec_state_lock_ to a reader-writer lock
Date Sat, 18 Mar 2017 00:31:41 GMT


Sailesh Mukil resolved IMPALA-4456.
    Resolution: Won't Fix

Need to rethink this.

> Change query_exec_state_lock_ to a reader-writer lock
> -----------------------------------------------------
>                 Key: IMPALA-4456
>                 URL:
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.7.0
>            Reporter: Sailesh Mukil
>            Assignee: Sailesh Mukil
>              Labels: performance, scalability
> The process wide query_exec_state_map_lock_ can be a highly contended lock. It is used
as a mutex currently.
> Changing it to a reader-writer lock and using it as a writer lock strictly only when
necessary can give us some big wins with relatively less effort.
> On running some tests, I've also noticed that changing it to a RW lock reduces the number
of clients created for use between nodes. The reason being ReportExecStatus() that runs in
the context of a thrift connection thread, waits for fairly long periods of time for the query_exec_state_map_lock_
on high load systems, causing the RPC sender to be blocked on the RPC. This in turn requires
other threads on the sender node to create new client connections to send RPCs instead of
reusing old ones, which contribute to nodes hitting their connection limit.
> So, this relatively small change can give us wins not just in terms of performance, but
scalability too.

This message was sent by Atlassian JIRA

View raw message