carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CARBONDATA-287) Save the sorted temp files to multi local dirs to improve dataloading perfomance
Date Sat, 08 Oct 2016 15:56:20 GMT

    [ https://issues.apache.org/jira/browse/CARBONDATA-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558202#comment-15558202
] 

ASF GitHub Bot commented on CARBONDATA-287:
-------------------------------------------

GitHub user QiangCai opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/217

    [CARBONDATA-287]Using multi local directorys to improve dataloading perfomance

    **1  Analysis**
    
    Now for each dataloading, we use only one different local directory to save the sorted
temp files. I think it is neccessary to use multi local directorys in multi disks for each
dataloading to improve dataloading performance.
    
    **2 Solution**
    
    Modify SortDataRows class to separate the sorted temp files into different local directorys
in different disks.. 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/QiangCai/incubator-carbondata localDirsWorkLoadBalance

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/217.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #217
    
----
commit 4d0a276bb7283dc0683efe9f3684fdff4cadf729
Author: c00318382 <c00318382@huaweiobz.com>
Date:   2016-10-08T12:54:32Z

    localdirs

commit 41c027833c19c3f3331470f48759f469159eb59c
Author: QiangCai <qiangcai@qq.com>
Date:   2016-10-08T15:26:39Z

    clean temp folder

----


> Save the sorted temp files to multi local dirs to improve dataloading perfomance 
> ---------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-287
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-287
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: data-load
>    Affects Versions: 0.2.0-incubating
>            Reporter: QiangCai
>            Assignee: QiangCai
>            Priority: Minor
>             Fix For: 0.2.0-incubating
>
>
> Now for each dataloading, we use only a different local dir to save  the sorted temp
files. I think it is neccessary to use multi local dirs for each dataloading to improve dataloading
performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message