carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xuchuanyin <...@git.apache.org>
Subject [GitHub] carbondata pull request #1199: [CARBONDATA-1335] Remove duplicated time-cons...
Date Thu, 27 Jul 2017 07:09:44 GMT
GitHub user xuchuanyin opened a pull request:

    https://github.com/apache/carbondata/pull/1199

    [CARBONDATA-1335] Remove duplicated time-consuming method call

    # Scenario
    
    Currently we did a concurrent  14 queries on Carbondata. The queries are the same, but
on different tables. We have noticed the following scene:
    
    + A single query took about 5s;
    + In concurrent scenario, each query took about 15s;
    
    By adding checkpoint in the log, we found that there was great latency in starting query
jobs in spark.
    
    # Analysts
    
    When we fire a query, Carbondata firstly do some job in the client side, including parse/analyze
plans and prepare filtered blocks and inputSplits. Then Carbondata start to submit query job
to spark. 
    
    We found in the first step, Carbondata took about 7s in current scenario, but it only
took about <1s in single scenario.
    By studying the related code, we found the most time consuming method call was  `CarbonSessionCatalog.lookupRelation`.
In side this method, it called `super.lookupRelation` twice, which consumed about 3s each
time.
    
    # Solution
    
    Carbondata only needs to call the `super.lookupRelation` only once, we need to remove
the useless duplicated method call.
    
    I've tested in my environment and it works well. In concurrent scenario, each query takes
about 12s (3s saved for the improvement).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata remove_duplicated_lookup

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1199.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1199
    
----
commit d05d1fadbe0f773a00ff1d0e96ff9fe90b7b7f06
Author: xuchuanyin <xuchuanyin@hust.edu.cn>
Date:   2017-07-27T07:07:25Z

    Remove duplicated time-consuming method call

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message