kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "fengYu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-2929) speed up Dump file performance
Date Wed, 24 Jan 2018 12:52:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16337544#comment-16337544
] 

fengYu commented on KYLIN-2929:
-------------------------------

Sorry for my bad memory, I will do this review this week. 

> speed up Dump file performance
> ------------------------------
>
>                 Key: KYLIN-2929
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2929
>             Project: Kylin
>          Issue Type: Bug
>          Components: Query Engine
>    Affects Versions: v2.0.0
>            Reporter: fengYu
>            Assignee: fengYu
>            Priority: Major
>              Labels: Performance
>         Attachments: 0002-KYLIN-2929-speed-up-dump-performance-write-dump-file.patch
>
>
> when I work on KYLIN-2926, I find coprocessor will dump to disk once estimatedMemSize
is bigger than spillThreshold, and found that spill data size is extraordinary smaller that
estimatedMemSize, in my case dump file size is about 8MB and spillThreshold is setting to
3GB.   
> So, I try to keep the spill data in memory rather than write the file to disk immediately,
and when those in-memory spill data reach the threshold, write all spill files together.
> In my case, the coprocessor process cost time drop from 22s to 16s, it is about 30% upgrade。



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message