impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IMPALA-5677) Consider ways to reduce the accumulation of clean pages when executing large spilling queries
Date Tue, 18 Jul 2017 15:41:00 GMT
Tim Armstrong created IMPALA-5677:
-------------------------------------

             Summary: Consider ways to reduce the accumulation of clean pages when executing
large spilling queries
                 Key: IMPALA-5677
                 URL: https://issues.apache.org/jira/browse/IMPALA-5677
             Project: IMPALA
          Issue Type: Sub-task
          Components: Backend
    Affects Versions: Impala 2.10.0
            Reporter: Tim Armstrong
            Assignee: Tim Armstrong


By design, the new buffer pool aggressively uses available memory to hold onto spilled data
in memory.

On some stress tests runs (and in other testing) I've seen the clean pages memory blow up
and cause Impala to use more memory than one might expect based on the query memory limits
(but not more than the process limit, so we're not going outside of the configured resource
constraints). 

I think this may hurt performance more than it helps in many cases because it will squeeze
out the OS buffer cache.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message