hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10092) LLAP: improve how buffers are locked for split
Date Tue, 31 Mar 2015 18:26:54 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389103#comment-14389103
] 

Sergey Shelukhin commented on HIVE-10092:
-----------------------------------------

This is not a simple problem... 

> LLAP: improve how buffers are locked for split
> ----------------------------------------------
>
>                 Key: HIVE-10092
>                 URL: https://issues.apache.org/jira/browse/HIVE-10092
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>
> Right now, for simplicity, entire split of decompressed buffers is locked in cache, in
case some buffers are shared between RGs, to avoid dealing with situations where we uncompress
some data, pass it on to processor for RG N, then processor processes and unlocks it, and
before we can pass it on for RG N+1 it's evicted. 
> However, if split is too big, and cache is small, or many splits are processed at the
same time, this can result in a deadlock as entire cache is locked. We need to improve locking
to be more granular and probably also try to avoid deadlocks in general (bypass cache?)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message