hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harish Butani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4963) Support in memory PTF partitions
Date Fri, 09 Aug 2013 17:40:51 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735038#comment-13735038
] 

Harish Butani commented on HIVE-4963:
-------------------------------------

We already do this. The rows are accumulated in a ByteBasedList; when it fills up it is spilled
to disk and a new ByteBasedList is added. So if there are less than 32Mb bytes needed(or whatever
is set by the user), there is no I/O.
The saving here comes from not holding the objects in a serialized form. Currently every field
access goes through deserialization. InMemoryPartition was going to be the case where the
user guarantees that there is enough memory so we just hold the deserialized objects. Am working
on a Caching wrapper on the PTFPartition which would hold onto deserialized objects, but is
backed by the serialized bytes in case we run out of memory.
 
But yes it would be nice to merge these 2 concepts into one thing. There is an overhead in
Caching over InMemoryPartition: at least an extra serialization, potentially more in both
time and space. But the overhead may not matter that much. Give me a couple of days to work
through this..
                
> Support in memory PTF partitions
> --------------------------------
>
>                 Key: HIVE-4963
>                 URL: https://issues.apache.org/jira/browse/HIVE-4963
>             Project: Hive
>          Issue Type: Bug
>          Components: PTF-Windowing
>            Reporter: Harish Butani
>         Attachments: HIVE-4963.D11955.1.patch
>
>
> PTF partitions apply the defensive mode of assuming that partitions will not fit in memory.
Because of this there is a significant deserialization overhead when accessing elements. 
> Allow the user to specify that there is enough memory to hold partitions through a 'hive.ptf.partition.fits.in.mem'
option.  
> Savings depends on partition size and in case of windowing the number of UDAFs and the
window ranges. For eg for the following (admittedly extreme) case the PTFOperator exec times
went from 39 secs to 8 secs.
>  
> {noformat}
> select t, s, i, b, f, d,
> min(t) over(partition by 1 rows between unbounded preceding and current row), 
> min(s) over(partition by 1 rows between unbounded preceding and current row), 
> min(i) over(partition by 1 rows between unbounded preceding and current row), 
> min(b) over(partition by 1 rows between unbounded preceding and current row) 
> from over10k
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message