kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-2929) speed up Dump file performance
Date Mon, 29 Jan 2018 08:42:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343075#comment-16343075
] 

Billy Liu commented on KYLIN-2929:
----------------------------------

Thanks [~feng_xiao_yu], but the patch merged failed. Could you rebase against the master branch
first? 

> speed up Dump file performance
> ------------------------------
>
>                 Key: KYLIN-2929
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2929
>             Project: Kylin
>          Issue Type: Bug
>          Components: Query Engine
>    Affects Versions: v2.0.0
>            Reporter: fengYu
>            Assignee: fengYu
>            Priority: Major
>              Labels: Performance
>             Fix For: v2.3.0
>
>         Attachments: 0001-KYLIN-2929-speed-up-dump-performance-write-dump-file.patch
>
>
> when I work on KYLIN-2926, I find coprocessor will dump to disk once estimatedMemSize
is bigger than spillThreshold, and found that spill data size is extraordinary smaller that
estimatedMemSize, in my case dump file size is about 8MB and spillThreshold is setting to
3GB.   
> So, I try to keep the spill data in memory rather than write the file to disk immediately,
and when those in-memory spill data reach the threshold, write all spill files together.
> In my case, the coprocessor process cost time drop from 22s to 16s, it is about 30% upgrade。



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message