impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Jacobs (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (IMPALA-5623) lag() on STRING cols may hold memory until query end
Date Tue, 11 Jul 2017 15:42:00 GMT

     [ https://issues.apache.org/jira/browse/IMPALA-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matthew Jacobs resolved IMPALA-5623.
------------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

commit 3939591aa891ce8e277f9c9a11e155927d413ea1
Author: Matthew Jacobs <mj@cloudera.com>
Date:   Thu Jul 6 17:13:03 2017 -0700

    IMPALA-5623: Fix lag() on STRING cols to release UDF mem
    
    IMPALA-4120 fixed an issue where lead/lag was potentially
    operating on memory that the UDA didn't own, resulting in
    potentially wrong results. As part of that fix, lead and lag
    started allocating 'global' UDF memory (e.g. via Allocate()
    rather than AllocateLocal()) in Init() which needs to be freed
    in Serialize() or Finalize(), but only lead() was updated to
    free the memory. This memory is eventually freed when the
    fragment is torn down, but as a result of not freeing the
    memory in Serialize or Finalize, the memory may be allocated
    longer than necessary.
    
    Change-Id: Id2b69b4ccb9cac076abca19bed6f0b1dd11dfff3
    Reviewed-on: http://gerrit.cloudera.org:8080/7371
    Reviewed-by: Matthew Jacobs <mj@cloudera.com>
    Reviewed-by: Dan Hecht <dhecht@cloudera.com>
    Tested-by: Impala Public Jenkins


> lag() on STRING cols may hold memory until query end
> ----------------------------------------------------
>
>                 Key: IMPALA-5623
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5623
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.8.0
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: Impala 2.10.0
>
>
> IMPALA-4120 fixed an issue where lead/lag was potentially operating on memory that the
UDA didn't own, resulting in potentially wrong results. As part of that fix, lead and lag
started allocating memory in Init() which needs to be freed in Serialize() or Finalize(),
but only lead was updated to free the memory. This memory is eventually freed when the fragment
is torn down, but as a result of not freeing the memory in Serialize or Finalize, the memory
may be allocated longer than necessary.
> A warning is printed when this happens:
> {noformat}
> [localhost:21000] > select concat(' foo ', lag(string_col,1,NULL) over (partition
by bool_col order by id)) from functional.alltypestiny order by id;
> Query: select concat(' foo ', lag(string_col,1,NULL) over (partition by bool_col order
by id)) from functional.alltypestiny order by id
> Query submitted at: 2017-07-06 13:56:24 (Coordinator: http://mj-desktop.ca.cloudera.com:25000)
> Query progress can be monitored at: http://mj-desktop.ca.cloudera.com:25000/query_plan?query_id=124dfe18a6cee76a:fafdea4000000000
> +----------------------------------------------------------------------------------------+
> | concat(' foo ', lag(string_col, 1, null) over (partition by bool_col order by id asc))
|
> +----------------------------------------------------------------------------------------+
> | NULL                                                                              
    |
> | NULL                                                                              
    |
> |  foo 0                                                                            
    |
> |  foo 1                                                                            
    |
> |  foo 0                                                                            
    |
> |  foo 1                                                                            
    |
> |  foo 0                                                                            
    |
> |  foo 1                                                                            
    |
> +----------------------------------------------------------------------------------------+
> WARNINGS: UDF WARNING: Memory leaked via FunctionContext::Allocate() or FunctionContext::AllocateLocal()
> Fetched 8 row(s) in 0.12s
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message